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Administrator 

National Aeronautics and Space Administration 
Washington, DC 20546 


Dear Mr. Goldin: 

The Aerospace Safety Advisory Panel is pleased to submit its annual report for 1998. 

Excellent relations between the Panel and NASA and its contractors characterized the past year. We saw 
exceptional safety consciousness wherever we went. The experience level and technical expertise of the 
government and private workforce remains impressive. Safety in the short term is well served. 

The long-term picture is less certain. NASA’s success depends heavily on the quality of its workforce. 
Unfortunately, much of the present workforce is nearing the end of its career, and there appears to be insuf- 
ficient succession planning. Tight budgets have forced severe limitations on hiring. It is unclear how the 
expertise will be developed to continue existing programs safely and effectively guide new efforts. 

Budget and external constraints have also forced a short planning horizon with respect to upgrades for the 
Space Shuttle and International Space Station (ISS). Even though both programs are presently operat- 
ing at an acceptable level of risk, there are identified improvements that could make them even safer. 
When these efforts are delayed, a valuable risk reduction opportunity is lost. 

NASA must also acknowledge the continuing, vital role of the Space Shuttle until well into the next cen- 
tury and develop plans accordingly. The reliance of the ISS on the Space Shuttle must be taken into 
account when considering its replacement with any new, human-rated Reusable Launch Vehicle. 


Sincerely, 

Richard D. Blomberg 
Chair 

Aerospace Safety Advisory Panel 
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T his report covers the activities of the Aerospace Safety Advisory Panel 
(ASAP) for calendar year 1998 — a year of sharp contrasts and significant 
successes at NASA. The year opened with the announcement of large work- 
force cutbacks. The slip in the schedule for launching the International 
Space Station (ISS) created a 5-month hiatus in Space Shuttle launches. This slack 
period ended with the successful and highly publicized launch of the STS-95 mission. 
As the year closed, ISS assembly began with the successful orbiting and joining of the 
Functional Cargo Block (FGB), Zarya, from Russia and the Unity Node from the 
United States. 

Throughout the year, the Panel maintained its scrutiny of NASA's safety processes. 
Of particular interest were the potential effects on safety of workforce reductions and 
the continued transition of functions to the Space Flight Operations Contractor. 
Attention was also given to the risk management plans of the Aero-Space 
Technology programs, including the X-33, X-34, and X-38. Overall, the Panel con- 
cluded that safety is well served for the present. 

The picture is not as clear for the future. Cutbacks have limited the depth of talent 
available. In many cases, technical specialties are “one deep.” The extended hiring 
freeze has resulted in an older workforce that will inevitably suffer significant depar- 
tures from retirements in the near future. The resulting “brain drain” could represent 
a future safety risk unless appropriate succession planning is started expeditiously. 
This and other topics are covered in the section addressing workforce. 

The major NASA programs are also limited in their ability to plan properly for the 
future. This is of particular concern for the Space Shuttle and ISS because these pro- 
grams are scheduled to operate well into the next century. In the case of the Space 
Shuttle, beneficial and mandatory safety and operational upgrades are being delayed 
because of a lack of sufficient present funding. Likewise, the ISS has little flexibility 
to begin long lead-time items for upgrades or contingency planning. For example, the 
section on computer hardware and software contains specific findings related to 
required longer range safety-related actions. 

NASA can be proud of its accomplishments this past year, but must remain ever vig- 
ilant, particularly as ISS assembly begins to accelerate. The Panel will continue to 
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focus on both the short' and long-term aipects of risk management and safety plan' 
ning. This task continues to he made mi nageable and productive by the excellent 
cooperation the Panel receives from both NASA and its contractors. Particular 
emphasis will continue to be directed to lcnger term workforce and program planning 
issues as well as the immediate risks associated with ISS assembly and the initial 
flights of the X'33 and XG4. 

Section II of this report presents specific findings and recommendations generated by 
ASAP activities during 1998. Section III :ontains more detailed information in sup' 
port of these findings and recommendatiois. Appendix A is a current roster of Panel 
members, consultants, and staff. Appendi < B contains NASA’s response to the find' 
ings and recommendations from the 1997 ASAP Annual Report. Appendix C details 
the fact'finding activities of the Panel in 1998. 

During the year, Mr. Richard D. Blombetg was elected chair of the Panel and Vice 
Admiral (VADM) Robert F. Dunn was elected deputy chair. VADM Bernard M. 
Kauderer moved from consultant to member. Mr. Charles J. Donlan retired from 
the Panel after many years of meritorious service. Ms. Shirley C. McCarty and 
Mr. Robert L. (“Hoot”) Gibson joined tP e Panel as consultants. 
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I. Findings and 
Recommendations 



FOR 1998 



A. WORKFORCE 

Safety is ultimately the responsibility of the crews, engineers, scientists, and techni- 
cians who, in collaboration with private-sector contractors, design, build, and 
operate NASA's space and aeronautical systems. The competency, training, and 
motivation of the workforce are just as essential to safe operations as is well-designed, 
well-maintained, and properly operated hardware. NASA has traditionally recog- 
nized this key linkage between people and safety by viewing its employees as “assets, 
not costs” and by sustaining highly innovative human resources initiatives to 
strengthen the NASA workforce. 

In recent years, a declining real budget has forced a significant downsizing of NASA 
personnel who manage, design, and process the Space Shuttle and the International 
Space Station (ISS) programs, especially at the Centers associated with human space 
flight: Kennedy Space Center (KSC), Johnson Space Center (JSC), and Marshall 
Space Flight Center (MSFC). To avoid a highly disruptive mandatory reduction-in- 
force (RIF), NASA has encouraged voluntary resignations through a limited 
“buyout” program, normal attrition, and a hiring freeze. This combination of ele- 
ments has been effective in avoiding an involuntary RIF, but it has not been able to 
avoid the consequential shortages in critical skills and expertise in some disciplines 
and capabilities. The transition of responsibilities from NASA to the United Space 
Alliance (USA) under the Space Flight Operations Contract (SFOC) has further 
affected the mix of duties and capabilities that are available to conduct NASA's day- 
to-day business associated with the Space Shuttle and the ISS. 

The problem is not limited to the Government workforce. Similar shortages of crit- 
ical skills resulting from the downsizing at USA have been noted in the NASA/USA 
Transition and Downsizing Review : Ground and Flight Operations , the Lang/Abner 
report of May 1998. 

Because KSC, JSC, and MSFC each face additional downsizing targets of 300 to 
400 positions by fiscal year (FY) 2000, the potential for additional shortfalls in key 
competencies clearly exists. Among other effects, the hiring freeze of the past several 
years has all but killed the usual pattern of bringing “new blood” into the Agency to 
replace those who are leaving through retirements, attrition, or voluntary 
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resignations. Although the hiring freeze lias now been lifted, budgetary restrictions 
make it all but impossible to replace experienced persons who are leaving. In these 
circumstances, the question of who wil be available and fully qualified to lead 
NASA’s human space flight programs in the post-2005 period has become real. In the 
shorter run, there are unanswered questions as to whether the combined workforce 
of NASA and USA will be sufficient to support an increased flight rate in the post- 
1999 period. This issue is also addressed i i the Space Shuttle section of this report. 

During this period, NASA has found it difficult to sustain its reputation as an agency 
that attracts and retains “the best and the brightest” among Federal employees. 
Recapturing this tradition will be an imfortant factor in NASA’s ability to sustain 
safe and successful future missions, as well as the vision required to sustain this coun- 
try’s leadership in space flight and aero-space technology. 


Finding #1 

Budget and personnel ceiling constraints on the hiring of engineers, scientists, and 
technical workers are moving NASA toward a crisis of losing the core competencies 
needed to conduct the Nation’s space flight and aerospace programs in a safe and 
effective manner. 

Recommendation #1 

Provide NASA’s human space flight Fi eld Centers, particularly KSC, JSC, and 
MSFC, with the budgetary resources and administrative flexibility needed to 
strengthen their human resource capabilities. 


Finding #2 

Shortfalls in workforce training within both NASA and USA, caused by downsizing 
and the related difficulty of hiring new people to fill skill shortages, can jeopardize 
otherwise safe operations. 

Recommendation #2 

NASA and USA should review critical stills training and certification requirements 
and institute programs to ensure the full proficiency of the workforce and the safety 
of the products being released. 




Finding #3 

The combined effect of workforce downsizing, the recent hiring freeze, and the 
SFOC transition, especially at KSC, has raised the possibility that NASA senior 
managers in the future will lack the necessary hands-on technical knowledge and in- 
line experience to provide effective insight of operations. 

Recommendation #3 

NASA should develop and promulgate training and career paths, with a special focus 
on providing hands-on technical knowledge and experience, so that NASA’s future 
senior managers will possess the range of skills and experience required for effective 
insight of the SFOC. 
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B. SPACE SHUTTLE PROGRAM 

Despite continuing long-term uncertainty s, the Space Shuttle system worked well in 
1998. The program took advantage of a reduced Space Shuttle flight rate to begin 
needed work on organization, infrastructure, and paperwork. Much remains to be done, 
however. In particular, with regard to the contractor workforce numbers and quality, 
there are no clear plans for accommodating an increased flight rate. This is a critical 
problem, which is treated in more detail in the “Workforce” section of this report. Also 
treated in the “Workforce” section but worthy of special note herein are two circum- 
stances that particularly affect the KSC w arkforce: the uncertainty of the manifest as 
managers wrestle with the ISS requirements and the long-term future of the Space 
Shuttle program as top management and politicians frame the future in terms of a 
Reusable Launch Vehicle (RLV). This, despite the fact that the Space Shuttle is essen- 
tial for the ISS and given the recent history of new-start acquisition programs, will 
probably be the first-line human space vehicle until well into the next century. 

Problems also exist because of the outmoded and limited computer hardware and 
software capabilities of the orbiter. Findings and recommendations related thereto 
are in the “Computer Hardware/Software 1 section of this report. There are also prob- 
lems looming in the area of logistics; insjfficient spares, workforce reductions, and 
the tenuous status of vendors for the program give pause for concern. On the other 
hand, the current systems, facilities, anc personal dedication and morale of those 
who are involved with the Space Shuttle on a day-to-day basis are not in question. 
In fact, the dedication of both Government and contractor personnel is of the high- 
est order. In addition, there is a well-established program of continuous improvement 
and continuous examination of critical processes throughout the system. Finally, 
there remain in place numerous interlot king processes, checks and balances, and 
individual knowledge, which will serve to stop work whenever the unusual arises or 
uncertainty prevails. In spite of the generally positive present conditions, the follow- 
ing findings and recommendations are relevant. 


Finding #4 

It is often difficult to find meaningful metrics that directly show safety risks or unsafe 
conditions. Safety risks for a mature vehicle, such as the Space Shuttle, are identifi- 
able primarily in specific deviations from established procedures and processes, and 
they are meaningful only on a case-by-case basis. NASA and USA have a procedure 
for finding and reporting mishaps and “cl )se calls” that should produce far more sig- 
nificant insight into safety risks than wot Id mere metrics. 

Recommendation #4 

In addition to standard metrics, NASA ;hould be intimately aware of the mishaps 
and close calls that are discovered, follow up in a timely manner, and concur on the 
recommended corrective actions. 




Finding #5 

A principal cause of Space Shuttle processing errors is incorrect documentation 
(“paperwork”). 

Recommendation #5 

NASA and USA must place increased priority on determining error sources, causes, 
and corrective actions for inadequacies in the documentation on which Space 
Shuttle processing is based and develop a management system that drastically reduces 
the time that it takes to incorporate paperwork changes. 


Finding #6 

While spares support of the Space Shuttle fleet has been generally satisfactory, repair 
turnaround times (RTAT’s) have shown indications of rising. Increased flight rates 
will exacerbate this problem. 

Recommendation #6 

Refocus on adequate acquisition of spares and logistic system staffing levels to pre- 
elude high RTAT’s, which contribute to poor reliability and could lead to a mishap. 


Finding #7 

NASA aircraft used for both Space Shuttle operations and astronaut training are 
increasingly out of date and, in several respects, may be approaching the unsafe. This is 
noticeably so in the case of the Shuttle Training Aircraft (STA) and the T-38 aircraft. 

Recommendation #7 

Continue to execute and accelerate as much as possible the current plans for the 
modernization and safety assessment of astronaut training aircraft. 


Finding #8 

The use of simulated Space Shuttle launch and flight operations for training and 
rehearsal has proven to be an effective technique for enhancing safety and efficiency 
and is especially valuable in the case of special or rarely performed procedures or after 
a long hiatus of effort. 

Recommendation #8 

Simulation-based training should be included in difficult or infrequent Space Shuttle 
operations whenever feasible. This type of training is especially needed after there 
has been a significant hiatus in performing an operation. 
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C. NTHWOTMUL SPACE STATION PROGRAM 

With the launch of the first two elements of the International Space Station, the 
program has entered the launch and integration phase. When, late in 1997, the 
launch schedules were changed to accommodate late deliveries of various elements, 
the ISS program took advantage of the delay to introduce Multi-Element Integrated 
Testing (MEIT). Subsequently, the Panel participated in an assessment of the total 
planned ISS test program. This test program was found to be satisfactory. The inclu- 
sion of MEIT provides for testing at a h gher level of integration than had been 
previously available. 

During the year, the ISS caution and warning system effort was expanded to include 
the beginnings of a true damage assessment and control component. The Panel will 
continue to monitor progress in this area as well as plans for ISS protection from 
micrometeoroids and orbital debris. 

Closely related to the ISS is the subject of extravehicular activity (EVA), which has 
acquired greater importance during the ISS construction phase. It is covered in a sub- 
sequent section of this report. 


Finding #9 

Some hardware is being used in MEIT before it has completed qualification testing. 
Software is also often used before its verification and validation is complete. In both 
cases, modification to the hardware or software may be required before certification 
is completed, thereby potentially invalidat ng the results of the initial MEIT testing. 

Recommendation #9 

When it makes sense to deliver hardware or software to system-level testing such as 
MEIT before qualification/certification is complete, the effect of any qualification- 
induced changes must be carefully evaluated for implications for regression testing. 
Final testing should always be run with validated software and qualified hardware. 


Finding #10 

MEIT is the highest level of integrated tes:ing available before committing ISS ele- 
ments to launch. To produce valid results, i his testing requires a high level of fidelity 
in emulators/simulators used in place of missing components. 

Recommendation #10 

The ISS program should ensure that high-fidelity simulations of on-orbit compo- 
nents are used in the MEIT and that the configurations of those simulators are 
validated to be in agreement with what has actually been orbited. 


Finding #1 1 

Astronaut crew participation in testing improves fidelity of the test and better famib 
iarizes the crew with systems and procedures. 

Recommendation #11 

NASA should continue to involve the crew in integration testing and do so more 
heavily and at an earlier stage. 
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Finding #12 

The current ISS requirement is for a single Crew Return Vehicle (CRV). Crew safety 
over the life of the ISS requires the availability on orbit of two CRV’s, each of which 
is capable of accommodating the entire crew. The Soyuz capsule, designated as the 
interim CRV, does not have a full crew capability. Also, it is uncertain that sufficient 
Soyuz capsules and their launchers will be available to supply the needs of the ISS. 

Recommendation it 12 

NASA should accelerate its program to develop and deploy two fulbcrew CRV’s and 
take whatever measures are necessary now to ensure the availability of sufficient 
Soyuz capsules and launchers until the CRV’s are ready. 


Finding #13 

Plans calling for availability on orbit in early 2003 of a U.S. CRV based on the X-38 
technology demonstrator are highly ambitious. Although much of the X-38 technob 
ogy is off the shelf, there are numerous features that rely on yet-unproven approaches. 

Recommendation #13 

NASA must not allow the limited CRV development time to compromise the con- 
duct of a thorough risk assessment and testing program. 


Finding #14 

In the ASAP Annual Report for 1997, the Panel expressed concern for the high 
doses of radiation recorded by U.S. astronauts during extended Phase I missions in 
Mir. Subsequent and continuing review of this potential problem revalidates that 
unresolved concern. The current NASA limit for radiation exposure is 40 REM per 
year to the blood-forming organs, twice the limit for U.S. airline pilots and four times 
the limit for Navy nuclear operators (see also Finding #23). 

Recommendation #14 

NASA should reduce the annual limit for radiation exposure to the blood-forming 
organs by at least one half to not more than 20 REM. 
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Finding #15 

By virtue of the several ongoing programs for the human exploration of space, NASA 
is pioneering the study of radiation exposure in space and its effects on the human 
body. Research that could develop and expand credible knowledge in this field of 
unknowns is not keeping pace with operat onal progress. 

Recommendation #15 

Provide the resources to support more comp letely research in radiation health physics. 


Finding #16 

Many deployable structures on the ISS anc satellites on which astronauts must work 
during EVA’s use pyrotechnic initiators. There is often no simple way for an EVA 
astronaut to know by visual inspection whether or not an initiator has fired when a 
structure has failed to deploy properly. 

Recommendation #16 

NASA should develop and require the use of pyrotechnic initiators that leave clear 
visual evidence that they have fired. These “fire-evident” initiators should he 
required for all applications that may be er countered by an EVA astronaut. 


Finding #17 

In the event that a primary crewmember is unable to fly on an assigned ISS mission, 
current plans call for substituting a crewmember from a backup crew. Backup 
crewmembers do not, however, train extensively with the primary crew. 

Recommendation #17 

If backup crewmembers are to be substituted individually to the primary crew, then 
those crews should conduct some meaningful degree of joint training. 


D. EXTRAVEHICULAR ACTIVITY (EVA) 

The intense period of EVA required for the on-orbit assembly and maintenance of 
the ISS focuses attention on the readiness of the total system, the astronauts and 
their support teams, and their unique equipment to perform safely and efficiently 
those arduous, complex, and potentially dangerous operations. The EVA Project 
Office is to be commended for the foresight to initiate early on the very constructive 
planning and training now in progress. This planning effort includes experienced 
astronauts, element contractors, and representatives of the program centers. The sue- 
cess of the very first EVA’s by the crew of Endeavour to mate Unity to Zarya is a 
testament to the quality of that planning and training. For example, in an effort to 
minimize the possibility for surprises on orbit, the specific tools and some 4,000 end 
items with which those tools must work were evaluated underwater in the Neutral 
Buoyancy Laboratory. 

The Panel finds, however, a number of deficiencies in EVA material, training, and 
policy that have the potential to disrupt the orderly process manifested over the next 
several years. There is a critical shortage of Extravehicular Mobility Units (EMU’s). 
The current inventory is very success oriented and does not allow for contingencies, 
such as damage to a suit on orbit. A similar shortfall exists in Simplified Aid for EVA 
Rescue (SAFER) flight units, both U.S. and Russian. 

The research and development of improvements to EVA assets has been severely cun 
tailed. Twenty-year-old technology in the current assets continues to stagnate. There 
is a need to design a spacesuit for the Mars mission as interest in that venture blos- 
soms. Improvements in radiation shielding on EMU’s are needed to protect 
astronauts during EVA without affecting their ability to perform assigned tasks. 
Finally, differences between U.S. and Russian training methods and operational poli- 
cies exist and must be resolved as soon as possible. 

In summary, the success of EVA will be crucial over the next several years. Resources 
should be made available now to correct deficiencies that might affect that success. 


Finding #18 

The EVA project lacks sufficient operational assets to meet unplanned contingen- 
cies. There are no spare Extravehicular Mobility Units (EMU’s). Only five U.S. 
Simplified Aid for EVA Rescue (SAFER) flight units will be available to meet a 
requirement to maintain three units on orbit. In addition, only four Russian SAFER 
units are planned. 

Recommendation #18 

To meet contingencies that are almost certain to arise, additional EMUWnd SAFER 
units or their critical long lead components should be procured as soon as possible. 
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Finding #19 

The three available sizes of EMU planar Hard Upper Torso (HUT) units will accom- 
modate crewmembers from the 40th percentile female to the 95th percentile male. 
Assumptions were made regarding the abi ity of crewmembers to upsize or downsize 
to fit the three available HUT sizes and operate safely and effectively in them. 

Recommendation #19 

To validate the ability of crewmembers to actually use the various available HUT 
sizes, crewmembers in each of the several s .ze combinations/configurations should be 
required to perform normal and emergency functions in training mockups to demon- 
strate that full capability is available to each. 


Finding #20 

The EVA Research and Technology (R&T) program has been highly successful, and 
its products have led to the development of significant safety and operational 
improvements to EVA hardware and procedures. Current funding for advanced R&T 
for EVA is extremely limited. 

Recommendation #20 

Restore the EVA R&T program to a leve that will permit further development of 
not only near-term safety and operability improvements but also long-term products. 


Finding #2 1 

The safety implications of EVA training for U.S. and international partner astronauts 
in the Russian Hydrolab are not well understood. In particular, the implications of 
higher suit pressures and Russian bends pro ocols have not been thoroughly analyzed. 

Recommendation #2 1 

NASA should study the procedures used ir the Russian Hydrolab to determine their 
safety and monitor all Hydrolab testing when U.S. astronauts are involved. 


Finding #22 

There is an initiative to modify the prebreathe protocol for EVA operations on the 
ISS. The target is a 2-hour prebreathe fre m any pressure with the same or better 
bends risk than the protocol currently used in Space Shuttle operations. 

Recommendation #22 

Prior to authorizing any reduction in prebre ithe protocol for EVA on the ISS, NASA 
should conduct a study to ensure that there is no increase in the risk of bends asso- 
ciated with the special circumstances of the proposed new protocol. 


Finding #23 

The greatest potential for overexposure of the crew to ionizing radiation exists during 
EVA operations. Furthermore, the magnitude of any overexposure cannot be pre- 
dieted using current models. 

Recommendation #23 

NASA should determine the most effective method of increasing EMU shielding with- 
out adversely affecting operability and then implement that shielding for the EMU's. 


Finding #24 

EVA ground rule 4.3.2.12, u No Simultaneous EMU/Orlan ISS Extravehicular 
Activity," is constraining and reduces flexibility. 

Recommendation #24 

NASA should reexamine this ground rule and consider a criterion for selecting either 
an EMU or the Orlan suit for a particular EVA based on the specific requirements of 
the EVA or the specific crewmembers performing the EVA. 


Finding #25 

The NASA Standard Initiator (NS1) on a SAFER unit tested on STS-86 on October 
1, 1997, did not activate because of a marginal design of the activating power supply. 
As a result, the unit could not function. The certification testing for the firing cir- 
cuit did not identify the power supply inadequacy. Also, an inadequate NSI emulator 
was used for most of the original SAFER certification (qualification) and acceptance 
tests (see also Finding #14). 

Recommendation #2 5 a 

The design and implementation of flight systems critical to safety and mission suc- 
cess should, at least, provide redundancy for system startup. 

Recommendation #25b 

All NASA Centers should review the design requirements for reliable activation of 
the NSI and assure they are adequate to be communicated to their suppliers, espe- 
cially those who are responsible for the design of firing circuits. All designs currently 
using NSI’s should be reviewed to assure that the firing circuits are adequate and have 
been appropriately tested. 

Recommendation #2 5c 

Qualification tests of safety-critical equipment must use flight-quality hardware. Any 
exceptions must require high-level program approval. 
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E. AERO-SPACE TECHNOLOGY 

NASA’s Aero-Space Technology Enterprise activities continue to be impressive. As 
befits the Nation’s premier aeronautics organization, the NASA aeronautics Field 
Centers, guided by the vision of the Three JhUars for Success Strategic Plan, lead the way 
in identifying enhancements for greater f ight safety in all environments. Likewise, 
those same Centers continue to make major contributions to aviation and space trans- 
portation technology and stimulate other: to do the same. Especially noteworthy is 
the pervasive evidence of enthusiastic dedication to safety on the part of every indi- 
vidual, even as development and testing explore the outer reaches of the unknown. 

Key to that dedication to safety is involved leadership at all levels. That leadership 
is committed to and backed up by outstanding training and solid procedures. 
Principal among the latter are those set forth in the Annual Operating Agreement 
between each Center and the Headquarters Office of Safety and Mission Assurance. 
Significantly, at aeronautics Centers from which research aircraft are flown, there is 
a strong and viable Airworthiness and Flight Safety Review Board ( AFSRB) process, 
including the involvement of top management as the ultimate authority. Those same 
Centers are wholeheartedly committed to a strong Flight Readiness Review process 
as a final preflight safety check. 

In addition to the impressive current fligh: safety approaches described above, many 
other ongoing efforts will also serve to enhance aviation safety if pursued to their 
conclusion. Among them are integrated vehicle health management, intelligent 
flight controls, winter runway friction research, synthetic vision, tile research, and 
the various flight research programs. 

As impressive as the Aero-Space Technology Enterprise is, there are some issues 
worthy of reflection. Among them i; the need for the Federal Aviation 
Administration (FAA) to play a greater role in the pursuit of the safety goals embed- 
ded in NASA’s Three Pillars. An encoui aging step in this direction is the recent 
signing of a memorandum of understandir g between the FAA and NASA. In addi- 
tion, both the X-33 and X-34 programs a: e quite ambitious and have the potential 
for safety problems if ambition is not tempered with appropriate analysis and testing. 
Finally, while other agencies, often the U. 3. Air Force, have responsibility for range 
safety when new vehicles are tested, NASA oversight of planning and procedures for 
such testing is essential. That responsibilit y cannot be abdicated. 
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Finding # 26 

Achieving the objectives of the first of NASA’s Three Pillars, Global Civil Aviation, 
requires greater involvement and support by the Federal Aviation Administration 
(FAA). 

Recommendation #26 

NASA should pursue further commitment from the FAA to participate in the first 
of NASA’s Three Pillars, Global Civil Aviation. 
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Finding #27 

The X-34 technology demonstrator program faces safety risks related to the vehicle’s 
separation from the L4011 carrier aircraft and to the validation of flight software. 
Moreover, safety functions seem to be distributed among the numerous contractors, 
subcontractors, and NASA without a clear definition of roles and responsibilities. 

Recommendation #27 

NASA should review and assure that adequate attention is focused on the potentially 
dangerous flight separation maneuver, the thorough and proper validation of flight 
software, and the pinpointing and integration of safety responsibilities in the X-34 
program. 


Finding #28 

Because XG3 and X'34 flight range safety is the responsibility of another agency, 
NASA may have a tendency to pay less attention to that aspect of the programs. 

Recommendation #28 

When NASA'Sponsored vehicles are using a test range, NASA should not abdicate 
its responsibilities to ensure safe flight. 
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F. COMPUTHI HARDWARE/80FTWMRE 

Computer systems continue to play an increasingly safety-critical role in NASA’s 
activities. They also represent areas of potential risk to major programs. Because of 
the vital importance of computer hardwire and software to NASA’s activities, the 
Panel has continued its practice of including a separate section on this topic. 

During this past year, the Panel has observed substantial progress related to computer 
hardware and software. The International Space Station program has instituted a 
schedule review activity that should provide an early warning of specific areas in 
which software development and test schedules are in danger. The ISS has also 
improved its software configuration mana >ement and is making deliveries of certified 
flight software. The new Checkout and Launch Control System (CLCS) at KSC is 
making good progress and promises to be a significant improvement over the exist' 
ing Launch Processing System (LPS). In addition, the Independent Verification and 
Validation (IV&V) facility in Fairmont, West Virginia, has made excellent strides in 
defining its mission and has shown substantial progress. 

Nevertheless, there are still a number of concerns for which action is needed. These 
are covered by the findings and recomme idations presented below. 


Finding #29 

The Space Shuttle General Purpose Computers (GPC’s) are outmoded and limit the 
ability to incorporate necessary software changes and hardware upgrades. 

Recommendation #29 

NASA should begin the process of replaci ig the Space Shuttle GPC’s. As part of this 
effort, NASA should also modularize the flight software. 


Finding #30 

There is no formal requirement that dependent Space Shuttle Lloads be recalculated 
or checked when an Idoad patch is to be jplinked. 

Recommendation #30 

NASA should create a dependency matrix of all Lloads. Furthermore, it should assess 
its Space Shuttle and ISS procedures and ensure that they are all fully documented. 


Finding #3 1 

Present plans depend on human procedures to achieve lockout to prevent inadver- 
tent or unauthorized access to actual hardware when using the new Checkout and 
Launch Control System (CLCS). 

Recommendation #31 

NASA should use a computerized authorization to achieve lockout of commands to 
actual hardware from anyone not authorized to issue such a command in CLCS. 
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Finding # 32 

NASA does not have a plan in place to deal with the problem of maintaining the 
many commercial off-the-shelf (COTS) software development tools used in its pro- 
grams. 

Recommendation #32 

NASA should develop a general strategy and provide programwide guidelines for 
addressing the maintenance of COTS tools. 


Finding #33 

The planning process for computer upgrades for the ISS has begun. Several possible 
upgrades are being discussed, such as replacing the Mass Memory Unit, upgrading the 
processor, upgrading the compiler used, and replacing the Portable Computer 
Systems (PCS’s). 

Recommendation #33 

NASA should proceed with the upgrade of ISS computer components expeditiously. 
In particular, the replacement of the mass storage device with solid-state memory 
should be made as soon as possible. 


Finding # 34 

Configuration management of ISS software does not include the source code for all 
of the elements being developed by the international partners. 

Recommendation #34 

NASA should strengthen the configuration control for ISS software to include soft- 
ware (source code as well as binary) and simulations produced by all international 
partners and vendors. 
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Finding #35 

The ISS presently has no programwide software development standards to manage soft' 
ware activities performed by NASA, its contractors, and the international partners* 

Recommendation #35 

The ISS program should establish prograi iwide standards to aid in specifying, design- 
ing, developing, and managing all future ISS software projects. These standards can 
be as simple as a set of best practices. 


Finding #36 

Several software developments are on the critical path for launch and operation of 
the ISS. While some software elements have had the early involvement of a multi- 
disciplinary team that includes users and operators, many have not. The lack of user 
involvement results in increased schedule and safety risk to the program. 

Recommendation #36 

The ISS program should follow a concur *ent engineering approach to building soft- 
ware that involves users and other key discipline specialists early in the software 
development process to provide a full range of perspectives and improve the under- 
standing of requirements before code is developed. 


Finding #37 

The recent compromising of the Data Encryption System (DES) suggests that the 
ISS command uplink may not be sufficie Ttly protected. 

Recommendation #37 

NASA should engage the National Security Agency to conduct a thorough evalua- 
tion of the level of protection provided by the current system and proceed as rapidly 
as feasible with its plans for a more secu e encryption system for the ISS. Potential 
vulnerabilities of the ground elements of the system should also be assessed. 
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A. WORKFORCE 

Ref: Finding #1 

In its Workforce Restructuring Plan (February 1998), NASA takes note of the hiring 
freeze that “contributes to technical stagnation and organizational atrophy” (page 6). 
The plan goes on to point out that NASA “now has more S&Es [scientists and engi- 
neers] over the age of 70 than below the age of 25.” Other NASA managers have 
reported to the Panel that there are twice as many engineers over age 60 than under 
30. The plan also reports that “NASA has been forced to virtually shut down its 
Cooperative Education Program — formerly one of the largest and most successful in 
the government” (page 6). This eliminates one of NASA’s major sources of new 
engineering talent. 

All NASA Centers now have authority to resume external hiring within budgetary 
ceilings. This is a positive step in the right direction. However, the shortfall of bud- 
getary resources also means that the Kennedy Space Center (KSC), Johnson Space 
Center (JSC), and Marshall Space Flight Center (MSFC) will find it all but impos- 
sible to resume the hiring of fresh talent until at least FY 2001. These Office of Space 
Flight Centers must continue to downsize their workforces to meet the targets set 
forth in the Zero Base Review (ZBR) conducted in 1995 and adjusted downward in 
subsequent budget reviews. This will mean an additional loss of 300—400 positions at 
each Center beyond the significant downsizing that has already been achieved (rang- 
ing between 17 percent and 30 percent of the workforce at each Center). However, 
the workload associated with the 1995 downsizing targets has changed significantly 
at these Centers (for example, the ZBR estimated that 1,500 persons would be 
assigned to the International Space Station (ISS); the actual number is close to 
2,500). Responsibilities associated with implementing ISO 9000 standards were also 
not included in the ZBR. Thus, the Centers have been hard pressed to reach the 
1995 and subsequent workforce targets and have found it difficult to maintain core 
skills and expertise. 
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NASA has initiated a “Core Capability Assessment” that holds promise for estal> 
lishing a more realistic human resources baseline that correlates with the projected 
workload at the Office of Space Flight Centers. This assessment will be completed in 
time to affect the FY 2000 budget process. In the meantime, NASA’s workforce 
deficit is likely to grow. Although it is theoretically possible to fill a documented crit- 
ical skills shortage, the Panel has found this is rarely done because of concomitant 
constraints on budget and personnel positions. As a consequence, managers are 
prone to “make-do” with a particular skills deficit or to look for a current employee 
who can be transferred or retrained. In some cases, this is an acceptable solution, but 
often the retraining cannot be accomplished in a satisfactory timeframe. The Panel 
is especially concerned that the normal patterns of bringing new technical and mam 
agerial leadership into NASA have beer seriously disrupted. Where it has been 
established practice to hire at least a few t ozen or so co-op students at each Center 
along with other outstanding “fresh-out” engineering graduates, these traditional 
channels of identifying NASA’s next generation of leaders have all but closed down. 

The proper response to the growing workforce crisis within NASA is not wholesale 
hiring but, rather, a steady accumulation of younger talent with critical skills and 
leadership potential in some reasonable relationship to the losses taking place 
through retirements and attrition. There a'e highly professional and creative human 
resources managers throughout NASA w 10 , if given an opportunity, can develop 
innovative strategies for strengthening NASA’s professional workforce. 

Ref: Finding #2 

The strategy of retraining and cross-training personnel to fill vacant positions caused 
by downsizing makes sense if there are suff cient resources and time to achieve certi- 
fied competency of the retrained or cr Dss-trained workers. There is evidence, 
however, that the combination of downsizing and transitioning duties under the 
Space Flight Operations Contract (SFOCl has resulted, in some cases, in less than 
satisfactory results. As noted in the Lang Abner report of May 1998, NASA/L/SA 
Transition and Downsizing Review (page 1 3), “An effective cross training program 
relies on a rigorous certification process that clearly defines proficiency and currency 
requirements. . . . This effort at cross-trair ing was felt by some of those interviewed 
to be hampered by downsizing (no time to do it). Some of those interviewed com- 
mented that people are being moved to ur familiar areas with minimal training.” 

In its conversations with technicians at k SC, the Panel’s KSC team was told that 
changing assignments under the SFOC fansition resulted in situations of uncer- 
tainty among technicians in which person tel were less likely to call “time out” than 
when a safety issue is clearly defined and understood by the responsible workers. The 
Lang/Abner report made a similar observ ition (page 7): “. . . feelings were mixed 
when asked if they were willing to say ‘stop’ or ‘time-out’ while performing a func- 
tion or releasing a product when they were not totally certain they understood 
the process or felt their products were ready.” Adequacy of training and rigor in 
certification is essential to eliminate these critical concerns. 



Annual Report 


Ref: Finding #3 

This is a restatement of an issue raised in last year’s ASAP Annual Report. 
Specifically, the Panel believes that effective insight by NASA under the SFOC 
must necessarily go beyond the traditional administrative skills associated with corn 
tract monitoring. The technical complexity of the Space Shuttle and the ISS means 
that no collection of data, in itself, will be sufficient to understand how the SFOC 
and its subcontractors are performing their operational responsibilities. The current 
generation of senior managers at KSC, JSC, and MSFC possess this depth of technn 
cal understanding because of their prior operational duties. However, as NASA’s role 
shifts from operations to oversight to insight and as obstacles continue in the orderly 
hiring of new employees, special efforts will be needed to provide the next genera^ 
tion of senior managers with the technical experience needed to achieve a real 
understanding of what is happening in Space Shuttle and ISS operations. Normal 
“career development programs” are not sufficient to provide such understanding. 
Special operational assignments and specific career paths will be necessary to develop 
a cadre of leaders for future senior management roles. An element of this process 
should be to ensure that a trained and qualified NASA presence is on the work floor 
even as operational duties transition from NASA to the SFOC. 

In short, as the SFOC transition proceeds, NASA will need to pay special attention 
to achieving and maintaining the technical competence of those senior managers who 
are charged with ensuring that the Space Shuttle and ISS are operated in a fully safe 
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B. SPACE SHUTTLE PROGRAM 

Ref: Finding #4 

KSC metrics published by the Office of Safety and Mission Assurance (OSMA) show 
charts such as time lost, exceedences of overtime restrictions, numbers of reworks 
required, reduction in the numbers of Government Mandatory Inspection Points 
(GMIPs), numbers of Incident/Error Review Board (IERB) incidents, and so forth, 
which are tools better suited to the program management of cost and quality than to 
safety. NASA’s safety management procedures demand detailed attention to the 
design, production, and procedures used ir. the Space Shuttle. Any deviations from 
these requirements would be of concern to safety, and must be evaluated on an indi- 
vidual basis. Metrics alone will not produce the required insight into safety. 

Subsystem managers at JSC do track safety -related concerns, but they do not neces- 
sarily report these as metrics. For example, there is an SFOC Incident Rate metric 
that is tracked and published, but it does not include any close calls wherein the 
potential for dollar damage is estimated to-be less than $1,000. 

“NASA Mishap Reporting and Investigating Policy” (NASA Policy Directive 
8621.1G) defines the procedures that NaSA uses to report and correct mishaps. 
OSMA’s Report of Space Shuttle Program/USA Process Review states that United Space 
Alliances (USAs) IERB will report corrective actions taken to address processing 
escapes, but leaves unclear what action NASA will take with this information. These 
data could produce far more meaningful measures of safety on the Space Shuttle pro- 
gram than metrics alone can provide. 

Ref: Finding #5 

Thousands of “deviations” and changes in the build paper and procedures used to pre- 
pare the Space Shuttle are waiting to be incorporated into the operational work 
paper. Metrics on workmanship errors indicate that the principal cause of such errors 
is “wrong” paper that is incorrect, incomplete, or difficult to understand. This has 
long been a problem in preparing the Space Shuttle for flight. Working with obso- 
lete paper is both inefficient and potentially hazardous to mission success. 

USA is developing some promising paperwork improvements, including the exten- 
sive use of graphics and digital photography to clarify the work steps, which should 
lead to increased safety and product qualiiy. The pace of developing these upgrades 
and incorporating them into the process p; per should be speeded up. A management 
system must also be developed that incorp crates these changes rapidly and reliably. 

Ref: Finding #6 

Problems requiring cannibalization contin ae. Two recent examples are the Ku-band 
deployed antenna assembly for STS-95 ard the continuing problem with the Mass 
Memory Unit (MMU). At the same time, the workload at the NASA Shuttle 




Logistics Depot (NSLD) is steadily increasing; this is the result of vendors and sup- 
pliers finding it uneconomical to further serve the program. Compounding it all are 
the demands of aging components and obsolescence, which are affecting shop work- 
load as it becomes necessary to perform more make or repair operations in-house. 
Recent staffing cutbacks at NSLD have exacerbated the problems. 

Throughout 1998, USA has conducted a continuing analysis of approximately 
80 items that presented difficulties with component and systems support. At the 
same time, the average length of component repair turnaround times has been 
steadily increasing. The rise is mainly associated with original equipment manufac- 
turers in their overhaul and repair practices, but it is also reflected in the NSLD 
effort. All these symptoms, of course, have been noted in a year wherein the launch 
rate was exceptionally low. In the 12 months commencing in May 1999, the Space 
Shuttle logistics system will be tested to the utmost. Therefore, it would seem pru- 
dent to resolve as many outstanding logistics issues as soon as possible. 

In resolving these outstanding logistics issues, it also must be considered that there are 
insufficient assets in the Space Shuttle program to support its expected life. The sup- 
port of the ISS will inevitably require the acquisition of further Space Shuttle 
assets — and not only reliance on innovative approaches to extending the life of exist- 
ing resources. 

Ref: Finding #7 

NASA aircraft used for astronaut training support include T-38’s and the Shuttle 
Training Aircraft (STA), a modification to the Gulfstream II. All are aging with no 
identified replacements. None have advanced nondestructive testing techniques 
available for helping to determine the actual life remaining. Of particular concern 
are the STA aircraft. They are rapidly approaching the end of their safe service lives, 
and no replacements are scheduled. The flight profiles required of these aircraft are 
more severe than those for which they were originally designed or certified. 
Furthermore, the capability to forecast and catalog the actual fatigue life expended is 
not precise. STA maintenance is excellent, but it cannot be expected to find every 
defect when operated in the required training environment. Plans to replace the STA 
with a newer aircraft need to be made. 

The age of the T-38 aircraft is of additional concern. The aircraft are vintage 1960’s, 
and no replacement is in sight. The installed ejection seat does not accommodate the 
full range of astronaut anthropometries and is potentially life threatening for those at 
the outer limits of reach and height. There is an ever-increasing need for time- 
consuming and expensive corrosion control. Bulkhead and inlet modifications are 
urgently needed, and a wing replacement program is under way. Engine nozzles may 
be in this same category. The installed avionics do not match those of the Space 
Shuttle orbiter and represent a serious training shortfall. 
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There are current plans to continue to upgrade the T-38’s so that their structural safety 
is maintained and they are equipped for f light in today’s air traffic control environ- 
ment. NASA also has a comprehensive p ogram to assure the safety of these aircraft 
before flight. It is essential that adequate funding for these efforts is maintained so that 
the present, excellent safety record with the training aircraft continues. 

Ref: Finding #8 

The use of simulation-based training is widely recognized as extremely important 
and, as a result, is fairly well institutionaliz id in the Space Shuttle program. Similarly, 
its use as a readiness verification and training tool is widely practiced in the aviation 
and nuclear power industries as well as the military. The value of simulation-based 
training is unquestioned in these arenas. It is noteworthy that simulation-based train- 
ing of Space Shuttle launch controllers in the wake of a long standdown was a 
significant factor in the smooth resumption of operations for STS-95. 

A wide variety of forms of simulated operations are in use for training, including neu- 
tral buoyancy facilities, the use of actual launch or flight control consoles with 
simulated data, and the use of computer-based simulations. Different levels of simu- 
lation fidelity (and corresponding differences in their developmental cost) have been 
used. Of particular interest are situations such as at KSC, where some of the actual 
operational hardware can be used in a trai ling mode because the incremental cost of 
simulation development in many such cases is minimal. NASA should thus consider 
the development and use of simulation-ba;ed training throughout the Agency in any 
difficult or complicated operation if an appropriate level of fidelity can be achieved 
at a reasonable cost. 
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Ref: Finding #9 

Multi-Element Integrated Testing (MEIT) was introduced into the ISS program in 
1997. MEIT has good potential for verifying compatibility among the several ele- 
ments of the ISS in ground-based tests before the hardware is delivered to the launch 
processors. When scheduling dictates it is prudent to deliver hardware to MEIT 
before qualification or other testing is complete, the potential is created for subse- 
quent hardware changes or repair that may invalidate the MEIT results. 

NASA and Boeing have developed an extensive array of testing procedures for inte- 
grated ISS software based largely on the use of simulations. At one end of the 
spectrum, Early Software Integration testing is used. To begin integration testing as 
soon as possible, software modules that are not fully validated are used in integrated 
testing with other components. By so doing, NASA and Boeing hope to uncover 
errors at an earlier stage than would be possible from usual techniques. While the 
technique has merit, its difference from more usual procedures has been a source of 
confusion — and possibly challenge — for various review groups. The point to be 
made, though, is that as long as full integration testing is performed with validated 
software later in the overall process, the technique is fine. Experience with this 
method has been very good so far. However, NASA needs to do a better job of artic- 
ulating how all parts of the process fit together. Also, final, full integration testing 
must always be performed with validated components. 

Ref: Finding #10 

A valid MEIT result requires that emulators/simulators correctly reflect the hardware 
and interfaces they substitute for in testing. Therefore, the simulations used should 
ultimately be certified or verified to behave exactly as their flight counterparts. Errors 
in the simulators could result in undetected errors in actual flight software. Thus, 
these simulators play as important a role in the overall process as does the actual 
flight software. Documentation reviewed by the Panel suggested that some testing 
might only be done with simulations of limited fidelity. 

Most testing performed at the Software Verification Facility (SVF) necessarily uses 
simulations for many components. The MEIT testing thus becomes the place in 
which the greatest amount of actual flight hardware can be included. Because not all 
hardware will be available at one time, including some that is already on orbit, sim- 
ulation must also be used with MEIT. 

It is important that when an element has been placed on orbit, it is replaced in MEIT 
by a validated simulation. In addition, it is important that as modifications are made 
to an element on orbit, such as correcting a latent software error or replacing a hard- 
ware unit with an upgraded unit, simulations are updated to reflect the current 
on-orbit configuration. 
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Ref: Finding #1 1 

Nothing better serves the accuracy and fidelity of testing than having the ultimate 
users — those whose lives will depend on the proper functioning of the system — 
involved in the testing. While test eng neers and others involved are certainly 
competent to validate designs and catch problems, astronaut involvement brings an 
additional dimension to the process. Also, such participation gives the crew an addi- 
tional opportunity to familiarize themselves with the systems — a familiarization that 
would become essential in the event of an emergency. It is recognized that available 
crew time can be scarce and expensive, bi t this should not prompt any tendency to 
avoid requesting astronaut involvement. Recent experience with 1SS hardware in 
the Space Station Processing Facility Fas highlighted the importance of crew 
involvement. It should be made routine. 

Ref: Findings #12 and #13 

The ISS currently has a requirement fo* a single Crew Return Vehicle (CRV). 
Consideration is being given to increasing this to two CRV’s once the ISS reaches 
assembly complete. There is a baselined location for one of the CRV’s on the Unity 
Node, and analyses are under way to ident fy the best position for a second vehicle if 
one is added. 

The X-38 is currently undergoing development at JSC, and a scaled version of the 
vehicle is being drop tested from the B 52 carrier aircraft at the Dryden Flight 
Research Center (DFRC). The X-38 concept for the ISS CRV is based on maximiz- 
ing the use of existing technology and off-' he-shelf equipment; however, the current 
design includes at least 15 unproved techrologies. For example, the use of a parafoil 
to gain a large cross-range capability must be certified for human use. The automated 
guidance and control system (including software codes) will have to be certified by 
extensive testing. The heat shield is another area of concern that will need extensive 
certification testing. In fact, the entire CR/ will have to be thoroughly analyzed and 
proved before the vehicle is fielded as the lifeboat” for the ISS. 

Plans are for the ISS program to issue a Request for Proposal (RFP) for a U.S. CRV 
in the spring of 1999. The planned RFP is basically for a design to the X-38 specifi- 
cation, although the responders will have fome latitude for changes. The target date 
for deploying the first CRV on orbit is March 2003, which is an ambitious schedule. 

Until a CRV is available, the plan is t ) use Soyuz capsules manufactured and 
launched in Russia. Each modified Soyuz b capable of returning three crewmembers. 
ISS mission rules dictate that a crew canrot be left on the station unless sufficient 
return capability for every crewmember is ivailable from an attached Space Shuttle 
or one or more CRV’s. 

The Panel conducted a review of CRV requirements for the Space Station Freedom, 
which was published as Appendix D to its Annual Report for 1992. That review con- 
cluded that there was a clear need to ha'e two CRV’s on orbit, each of which is 
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capable of returning the entire crew. There are several reasons why mission objectives 
dictate two CRV’s. The first and perhaps the most obvious is to limit the likelihood 
that a crewmember will be cut off from access to a CRV by a fire, toxic spill, or depres- 
surization event. To minimize this possibility, the two CRVs must be located at 
dispersed sites, which are chosen on the basis of expected crew positions during ISS 
operations. 

There are, however, at least two other equally compelling reasons for deploying two 
CRV’s. First, the availability of two-vehicles greatly increases the probability of 
having a functioning CRV when needed without forcing unrealistic or extraordinar- 
ily expensive reliability requirements on the design. Second, a two-vehicle 
deployment permits sending less than the full crew home in the case of a medical 
emergency. If only a single CRV were available, the entire crew would have to return 
when any one crewmember became seriously ill or injured. 

The current situation leads to several concerns. First, the plans to use the Soyuz as 
an interim CRV have been complicated by an uncertain delivery schedule. Each 
Soyuz only has an on-orbit life of 6 months. Therefore, the ISS depends on a con- 
tinuing flow of Soyuz capsules in a CRV configuration and sufficient Russian 
launcher capability until a U.S. CRV is ready. The Russians may be unable to meet 
a long-term ISS need for Soyuz return vehicles. Also, if the planned availability date 
of early 2003 for a CRV slips, the need for Soyuz replacements will grow, thereby fur- 
ther exacerbating the problem. 

In light of this situation, NASA must clearly accelerate the CRV development 
schedule as much as possible. This was also recommended by the Report of the Cost 
Assessment and Validation Task Force of the International Space Station (“Chabrow 
Report”), which specifically suggested combining the X-38 and CRV programs as 
soon as feasible to achieve the earliest possible CRV operational readiness date. The 
concern, however, is that in the haste to ready a CRV and avoid dependence on the 
Russians, there may be a tendency to omit important risk assessment and testing 
steps. An appropriate approach is needed that moves the U.S. CRV forward with all 
deliberate speed while doing everything possible to ensure Soyuz availability until its 
replacement is ready. 

Ref: Finding #14 

The field of radiation health physics is far from an exact science. For example, radi- 
ation detection and recording devices are recognized as less than adequate. Total 
exposure is not measured (for example, the neutron contribution is not recorded). 
Exposures of crewmembers who have performed similar on-orbit tasks and routines 
on the same flight vary considerably, casting doubt on the accuracy of the dosimetry. 
Models used to predict the exposures of crewmembers are discrepant. Certain 
space/solar events cause significant and unpredictable variations in the radiation 
field. In addition, the long-term effects of radiation on the human body (cancers and 
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genetics) lack a definitive understanding. All of these unknowns, plus others, should 
dictate a very conservative approach to controlling exposure to radiation. The gov- 
erning principle universally accepted in the nuclear business, from weapons 
production to power generation to mecical radiology, is “As Low As Reasonably 
Achievable” (ALARA). To that end, the U.S. domestic airlines limit annual crew 
exposure to 20 REM, and the Naval Nuclear Propulsion Program limits crew and 
workers to 5 REM per year and no more than 3 REM per quarter. The ISS, on the 
other hand, allows an exposure of 40 REM per year. 

Design or construction limitations in shit lding for ISS modules may be countered to 
some extent by well-planned procedures c nd routines. Considerations for minimizing 
radiation exposure should be better factored into ISS designs and operations. 

Ref: Finding #15 

There are many gaps in the relatively new field of science that examines the effects 
on humans of radiation exposure during space flight. For example, research is needed 
to establish the effects of linear energy transfer radiation, to determine the true con- 
tribution of neutrons to total radiation fi ;ld, to develop a model for the exposure of 
astronauts conducting an EVA during geomagnetic storms, and to calculate trans- 
mission functions under magnetically disturbed conditions. Other unknowns include 
such factors as the effects of secondary radiation from shielding, variances in dose 
from a “normal” state to solar maximum, the fact that current radiation limits are 
based on data derived from weapons acute exposure (whereas ISS crew exposure will 
be low dose rate), and efforts to improve the performance of personal dosimetry 
devices, among others. 

The investment in resources to expand re: earch in radiation health physics should be 
made now. In addition to helping the ISS, it would also provide an advance on a 
Mars mission. 

Ref: Finding #16 

NASA satellites and ISS elements male extensive use of pyrotechnic initiators 
(“pyros”) for such tasks as deploying strictures, releasing holddowns, and opening 
valves. When these work as intended, tf ey dissipate their energy in the process of 
performing their intended tasks. If they fa 1 to fire, however, they not only may leave 
a task uncompleted, but also create a pote itial hazard for an EVA astronaut. A visual 
inspection often cannot determine whether the pyro has fired. As a result, EVA 
crews cannot be sure from a visual inspection whether they face a potential hazard 
from unfired pyros. 

To increase safety when EVA crews mu. t work in the vicinity of pyros, a means 
should be developed to make the pyros “fire evident.” Simply, some visual method 
should be included with each pyro so that its firing will leave a conspicuous trace. For 
example, a small amount of inert dye might be added to the charge so that it leaves 



a distinctive color after it had been fired. This would make it easy for an astronaut to 
determine when pyros were still potentially live, thereby avoiding unnecessary risks. 
Once such a means is developed, its use should be mandated in all NASA applica- 
tions that might be encountered by an EVA astronaut. 

Ref: Finding #17 

A sense of teamwork and bonding is developed during the conduct of training on 
complex operations. Last-minute substitutions in a crew can disrupt that team envi- 
ronment, while the new member plays “catchup” with his new crew’s style. If backup 
crews participate in some significant training exercises with the primary crew, the 
potential for disruption can be minimized. 
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D. EXTRAVEHICULAR ACTIVITY (EVA) 

Ref: Finding #18 

The success of the 1SS program depends heavily on EVA assembly operations. The 
procurement cycle for Extravehicular Mobility Units (EMU’s) is about 2 years. 
Simplified Aid for EVA Rescue (SAFER) units, both U.S. and Russian, also require 
significant time to manufacture and qualify. The EVA project has been limited to 
acquiring only the minimum numbers currently planned. This is a very success- 
oriented inventory. Accidental damage or complete loss of EMU’s during the extensive 
assembly operations could compromise such a plan, SAFER units (with a 1-year 
certification) must be rotated to mainta n three units on orbit. Damage could pre- 
clude meeting that requirement. Sound risk management suggests the need to 
procure additional units and spare components. 

Ref: Finding #19 

Although the EMU sizes were selected based on anticipated crew size distribution, 
the Panel is concerned that at some corrbinations of anthropometry and capability, 
a crewmember might not be able to perform the complete range of normal and emer- 
gency operations. 

Ref: Finding #20 

Present EVA equipment and procedures are based on 20-year-old designs and tech- 
nology. While NASA has had a strong Research and Technology (R&T) program 
that has led to the identification of signif cant improvements in EVA equipment and 
procedures, cutbacks in the funding for the R&T program have curtailed the possi- 
bility of the development of efficiency and risk reduction improvements. There is a 
clear need for further advances to support EVA activities over the extended lifetime 
of the 1SS and beyond. Based on prior R*kT achievements, a relatively small invest- 
ment now has the potential to yield sign ficant risk and cost reductions over the life 
of the ISS. 

Ref: Finding #21 

The Russian Orlan suit operates at a higher differential suit pressure (5.8 psi) than 
that of the U.S. EMU, which operates i t a 4.3 psi differential. Thus, personnel in 
underwater training in the Russian Hydralab are at a significantly higher total pres- 
sure, with a resulting increase in susceptibility to the bends. In addition, the protocol 
used in the Hydrolab does not match that used in the U.S. Neutral Buoyancy 
Laboratory (NBL) as far as prebreathe and bends monitoring are concerned. Also, 
the Hydrolab does not use Nitrox, which is used in the NBL as an aid to reduce bends 
and increase allowable training time at depth. There are major differences in the 
training and safety environments between the two facilities. A thorough under- 
standing of these differences is required, md training safety should be monitored. 




Ref: Finding #22 

The long-standing Space Shuttle program prebreathe protocol of 4 hours (from a 
14.7-psia cabin) has proven to provide a minimal risk of bends. Any change to that 
protocol should be based only on credible empirical evidence. 

Ref: Finding #23 

ISS and Shuttle crews conducting EVA’s are at maximum risk for significant radia- 
tion exposure. It may not be possible to terminate critical operations during a 
radiation “alarm” condition. Additional shielding for the EMU’s would mitigate this 
risk. This is an example of crucial research that should be undertaken in view of the 
magnitude of the EVA tasks facing the ISS program during the assembly phase, as 
well as the need to protect the astronauts. 

Ref: Finding #24 

Because all international crews are trained to perform EVA’s in either the EMU or 
the Orlan suit, there is much flexibility possible in the scheduling of EVA operations. 
EVA ground rule 4.3.2.12 states: “The nominal plan for ISS EVA planning is to 
select either EMUs or Orlans for a particular increment.” This means that one or the 
other suit will be selected for an increment of several months and that all EVAs 
during that increment will use the designated suit without regard for the purpose of 
that particular EVA. This seems overly constraining and may not allow for the opti- 
mization of crew time on orbit. It also can restrict a crew working on their own 
portion of the ISS from using their own suits. 

Ref: Finding #25 

The SAFER system is intended to give an astronaut the capability to return to the 
parent vehicle in the event that the tether, normally used to prevent the astronaut from 
drifting away during an EVA, becomes disconnected. The functioning of the SAFER 
unit can therefore be critical to crew survival when an astronaut becomes untethered. 

A basic requirement imposed on the original design of the Space Shuttle was that, 
in the event of a series of failures in a critical system, redundancy must provide fail- 
operational, fail-operational, fail-safe capability. This requirement has been adhered 
to in the major hardware components and assemblies, but it was not applied to the 
design of the SAFER unit, in which a single NASA Standard Initiator (NSI) was pro- 
vided to open gas flow to the manifold. There is no redundancy provided to back up 
the function of the NSI and therefore no backup means of initiating the gas flow that 
the unit requires to operate. Zero fault tolerance may be appropriate for emergency 
equipment on the grounds that it is only needed after another failure has occurred, but 
such equipment surely is expected to be at least capable of being activated when 
required. Thus, redundancy for activation should be at least single fault tolerant. A 
SAFER failure such as the one that occurred in the test on STS-86 could have been 
a Criticality 1 failure if the equipment had been called on in a true emergency. 
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The detailed March 30, 1998, report on tie NSI failure (STS-86 USA Simplified Aid 
for EVA Rescue (SAFER) Failure — Mishi p Date : October I, 1997 — Failure Review 
Board Report) emphasizes that the design of the activating power supply did not take 
into account the increase of NSI resistance as power was applied to the device. This 
was a fundamental failure to recognize the nature of the device intended to be acti- 
vated by fusing a bridge wire. The documentation of the NSI on which the design was 
based is a “fly-sheet” specification, which does not supply sufficient detail of this sort 
nor specify the impedance versus supply voltage (or admittance versus supply current) 
for an adequate power supply. Clearly, the variation of the impedance with activating 
current was not included in the specification, nor was it considered in the design. 

Contributing to the failure to identify thu problem at an early date was the fact that 
an NSI was used in only two tests on the c ertification unit. For all other certification 
tests and all flight units, an inadequate emulator (a fixed resistance of the nominal 
unactivated value) was used. 

Because the NSI is used operationally throughout NASA, the circumstances leave 
open the possibility that other users may have not properly understood the require- 
ments for the design of the tiring circuitn. This warrants further examination. 




E. AERO-SPACE TECHNOLOGY 

Ref: Finding #26 

The President has directed that all efforts be made to reduce the aircraft accident 
rate, to reduce the emissions and perceived noise levels of aircraft, and, while main' 
taining safety, to triple the aviation system throughput and reduce the cost of air 
travel. NASA, through its “Three Pillars” approach, has defined a roadmap directed 
at achieving these goals, but they cannot be achieved without the total cooperation 
of industry, academia, and the Federal Aviation Administration (FAA). Arguably, 
the most important of these is the FAA; yet that agency has recently withdrawn sig' 
nificant funding — particularly from the safety efforts that NASA was pursuing 
cooperatively with the FAA. Unless the FAA renews its funding participation, the 
pursuit of Pillar One, Global Civil Aviation, could well come to naught, and the 
President’s directive will end up being forgotten. More importantly, the laudable and 
extremely important goals set for Pillar One will have a much reduced probability of 
being achieved. 

Ref: Finding #27 

Orbital Sciences Corporation (OSC), in accordance with a NASA contract and 
NASA funding, manages the X-34 program. OSC is principally the systems integra- 
tor, with more than 30 subcontractors involved. While OSC’s program and fiscal 
management seems sound, the same cannot be said for all facets of the X-34 safety 
program. Particularly worrisome is the apparent lack of adequate testing and analysis 
with regard to separation of the vehicle from the L- 1011 and legacy software. 

While wind tunnel tests simulating the separation of the X'34 from the HOI 1 have 
been successfully completed using scale models of the two vehicles, more test and 
analysis may be required. The X~34 release mechanism is based on the flight^proven 
Pegasus release mechanism designed by OSC, but it may still require more testing for 
safety assurance. The aerodynamic forces and flying qualities of the combined vehb 
cles will also be assessed during various prerelease test flights. 

For the X'34, intentions are to reuse software from other systems, such as the Space 
Shuttle and Pegasus. Experience in other situations has demonstrated conclusively 
that, past satisfactory performance notwithstanding, to avoid unsafe performance, 
legacy software must be subjected to rigorous verification and validation before oper- 
ational use. The X~34 flight software is scheduled to be carried through a thorough 
verification and validation testing process by OSC. Performance tests of the X'34 
navigation system hardware and software have already been conducted at the White 
Sands Missile Range using an aircraft platform. 
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Given the plethora of participants in the <-34 program, it is essential that a safety 
plan with clearly defined responsibilities, particularly for safety integration, 
be spelled out. Examples of currently existing gray areas are procurement and manu- 
facturing quality assurance, preflight test and flight test plans, and range safety 
(including flight termination). 

Ref: Finding #28 

The flight profiles for the first X-33 tests will originate in the Air Force Flight Test 
Center (Edwards) test range and are scheduled to end at Michael Army Airfield on 
Dugway Proving Grounds, Utah, which is a storage facility for chemical and biolog- 
ical weapons. Later tests will go from Edwards to Malmstrom Air Force Base in 
Montana. While these routes generally traverse unpopulated areas through estab- 
lished military corridors, they also cross several major highways and terminate near 
vulnerable areas. Also, should there be an unexpected flight termination, the impact 
could, conceivably, be in a more populated area. This is particularly true because the 
destruct mechanism depends on a hard-over flight control signal leading to an aero- 
dynamic breakup that could result in a rather large ground impact footprint. If this 
were to happen while unspent propellant is still aboard, the results could be disas- 
trous. Communications failure or command termination failure could exacerbate the 
situation. The Air Force is conducting the appropriate risk analyses, but NASA must 
play an integral role in these analyses and not abdicate any of that responsibility to 
the Air Force or any other agency. 

For the X-34, while a much shorter range vehicle, the flight termination system 
might necessarily be activated with unspen : fuel on board. Appropriate analyses seem 
to be under way, but, again, NASA must te involved. 




F. COMPUTER HARDVWRE/SOFTWARE 

Ref: Finding #29 

Throughout the history of the Space Shuttle program, there has been a continuing 
demand to improve the functionality and maintainability of the General Purpose 
Computer (GPC) system through changes to the GPC software. Virtually every flight 
sees some level of software revision. At longer intervals, major upgrades to the soft' 
ware take place. There has been a general tendency for the memory and processor 
requirements to grow during this process. There is now little growth capacity left. 
The software has also become extremely intricate and difficult to maintain because 
of the many changes and the lack of modularity. 

The current GPC is technologically obsolete, and maintenance issues can be 
expected to continue to increase. While NASA purchased a supply of replacement 
chips and components, there is a question of whether this supply can last the life 
of the Space Shuttle. The obsolescence applies not only to the hardware, but also 
to the software development techniques forced by the hardware. The software is far 
less modularized than current technology and good practice would indicate. This, 
in turn, makes it more difficult and expensive to make changes, perform testing, 
and ensure that there are no undesirable side effects. 

While there is not an imminent crisis in the Space Shuttle computer and avionics 
systems, there are three critical factors: 

1. The system is currently approaching its capacity limits. 

2. The time required for any major upgrade in computer/avionics hardware or 
redevelopment of the basic flight software can be very long. The last complete 
GPC upgrade took 8 years. 

3. The delay of risk-reducing upgrades that are inevitable only postpones their 
safety benefits. 

Ref: Finding #30 

T loads are constants that tailor the Space Shuttle flight software to a specific flight. 
There are thousands of these constants. Most are calculated and folded into a flight 
software load well ahead of time through an elaborate procedure that includes a 
number of checks to be as certain as possible that things are correct. Some, such as 
those related to day-of-launch winds, are uplinked only shortly before flight. The 
Hoads are not all independent values, however; some depend on others. 

An issue has arisen with respect to Hoads that is indicative of a much bigger prob- 
lem that should be addressed. There is no formal requirement that dependent Hoads 
be recalculated or checked when reabtime Hoad updates are performed via com- 
mand load uplinks. In Panel discussions on this matter, it was revealed that the 
dependencies, in many cases, are known only in the minds of specific individuals in 
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the program. There is no map of the dependencies so that someone else can ascertain 
what recalculations need to be done. While the requirements issue is being worked, 
there is no evidence that anyone is working on developing a dependency map. 

This is indicative of the broader issue of undocumented knowledge that exists in the 
heads of only a few individuals. The dimensions of this problem are not known. 
Although it could require a large effort o document the key procedural items for 
Space Shuttle and 1SS computer operations, it is crucial to do so. 

Ref: Finding #31 

An incident occurred recently with the launch Processing System (LPS) in which 
an operator thought he or she was interacting with a simulation in a test mode. In 
reality, the operator was connected to an actual vehicle. While there were no severe 
consequences, this incident points out that there is no nonprocedural LPS lockout to 
prevent this kind of inadvertent access to a real vehicle. At present, it is possible for 
an operator to mistakenly interact with a real vehicle when he or she intended to be 
in a different room interacting with a simt lation. In the new control room associated 
with the Checkout and Launch Control System (CLCS), the plan is to address this 
issue by requiring each person entering the control room to go to the flight or launch 
director and obtain authorization for an activity. Each person will then have to log 
in to the activity. The plan is to have the CLCS display the activity on the screen so 
that people can see what activity is being carried out. The avoidance of a recurrence 
of the incident noted in the finding is to he handled by these procedural means. 

The CLCS will allow the handling of two orbiters in Orbiter Processing Facilities 
from the same control room. In achieving this, a system is being built to keep the 
data streams completely separate at the i ser interface level. Something similar for 
distinguishing simulations from real activ ties and locking a user out from anything 
other than the activity for which the director has authorized him or her is needed. 
NASA is urged to try to incorporate this kind of safety capability into the CLCS. 

Ref: Finding #32 

There is a fundamental dichotomy betwet n the use of rapidly changing commercial 
off-the-shelf (COTS) software development tools and long-lived systems. In particu- 
lar, the compilers and software development tools that NASA uses for programs such 
as the ISS and the CLCS are a matter of concern. The principal difficulty with tool 
maintenance arises from the fact that the lifetime of the operational software is more 
than an order of magnitude greater than the update cycle on the COTS tools used for 
developing the software products. For exati pie, the ISS code is expected to be used for 
a decade or two, while the tools used to de velop it are often upgraded annually. 

This is a very important fundamental problem. If NASA uses a COTS product that is 
current at the time of a software change, a recertification of the tool would be neces- 
sary for virtually all program changes. If NASA tries to stay with the version of a 


COTS tool used for the basic development of the software, then NASA will probably 
have to assume the responsibility for maintaining the tool on the computer systems 
used because it is unlikely that the vendor will do so. Staying with older tool versions 
also means foregoing fixes and improvements incorporated in later versions by the 
supplier. Either choice is associated with many problems. While there has been dis- 
cussion of updating some of the tools, such as compilers, this should not be done until 
a strategy is developed for addressing the overall problem. Otherwise, NASA could 
incur unexpected costs, incompatibilities, and delays in delivering new software. 

To begin addressing the problems associated with the tools used, a “tools czar” has 
been appointed. This is an important role and must be afforded the resources and the 
authority to set policy that can maximize the value of the tools to the projects. 

Ref : Finding #33 

The Mass Memory Unit (MMU) currently being deployed on the ISS is a mechani- 
cal rotating device. There are serious concerns about its long-term reliability. 
Although this risk has been deemed acceptable, it is no longer necessary. An alter- 
native is to use flash memory technology. A prototype has already been built 
that would enable the replacement of the 300-megabyte mechanical units with 
500-megabyte solid-state units. The cost is relatively small. 

NASA also is currently studying the use of Pentium technology to replace the 
Multiplexer/Demultiplexer (MDM) central processing units. Presently, the MOBILE 
Pentium MMX technology has passed initial screening. The next step is to build an 
engineering version for testing with existing flight software. Efforts are also being 
focused on software tools and the development platform on which they run. 
Developing a long-term tool maintenance strategy should precede any compiler or 
platform change. 

The expected quantity and use of ISS onboard laptops makes it impractical to upgrade 
all Portable Computer Systems (PCS’s) simultaneously. Rather, NASA and Boeing 
expect to upgrade the laptops in an evolutionary manner, incorporating new technol- 
ogy laptops from time to time. They plan to make the first upgrade at flight 5A. The 
next hardware upgrade is planned for 2001, with successive upgrades at approximately 
5 -year intervals. This seems to be an appropriate strategy for planning and executing 
an ISS computer upgrade and might be used as a model for other ISS components. 

Ref: Finding # 34 

The configuration management of ISS software has been improved with Boeing’s 
adoption of a single-configuration management system for all software flowing 
through the Software Development and Integration Facility (SDIL) at JSC. There 
are still some concerns, however. The standardization of software configuration 
applies only to software developed by Boeing or that NASA and Boeing have entered 
into the SDIL. All software produced by vendors or outside of NASA and Boeing is 
not included in the configuration management. 
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Most notably, the source code produced bv the Russians for the Service Module (SM) 
does not come under configuration conti al, nor is it ever delivered to NASA. The 
executable code (compiled and linked code) comes under the configuration man^ 
agement system only when it has been delivered to the SDIL. 

In addition, there is concern regarding the SM simulation that is used in the SDIL 
and as part of MEIT at KSC to test software in systems that interface with the SM. 
The Russians supply this simulation software. The SDIL only gets the upgrades at 
infrequent intervals when the Russian pa tners bring new code to be integrated and 
tested. In between these points in time, the Russian partners update and use their 
updated simulations, while simulations in the United States are out of synchroniza- 
tion with the version the Russians are usiig. The coordination between upgrades to 
the SM simulation should be improved. 

Ref: Finding #35 

The ISS relies heavily on software for sn ooth, safe operation. The involvement of 
several countries in developing ISS software further complicates an already complex 
program. Currently, each software development organization and each international 
partner has separate standards for software development. This makes the general 
management of the program more difficult:. It also heightens the potential for sched- 
ule slips and for shortcuts that could jeopardize safety. A more standard approach to 
developing and delivering software should be adopted to avoid the typical pitfalls of 
large software projects. 

Areas to be included in standards and bes: practices are: 

• Proven processes for involving operators, crewmembers, and other users in the 
requirement specification, conceptual design, and test planning activities for the 
software 

• State-of-the-art processes and minimi m criteria for computer, programming lan- 
guage, and tool selection and maintei iance 

• Requirements for programmer training levels 

• Philosophies for testing, including stress and long-duration tests 

• Independent verification and validati m requirements 

• Configuration management controls 

• Simulation validation and maintenance approaches to ensure that simulators 
accurately represent the requirements ind reflect the current state of the software 

• Processes for maintaining and upgrading the software 

It is further recommended that the So : tware Engineering Institute’s Capability 
Maturity Model be used to assess the capability level of each participating organiza- 
tion to successfully deploy complex softwe re systems. 


Ref: Finding #36 

The development of complex 1SS software systems involves many disciplines and 
requires extensive experience with space systems. Project histories indicate that the 
broad involvement of users and key discipline specialists in the early phases of software 
development pays off in the increased understanding required for success. A recent 
study published in Communications of the ACM that reported on software projects in 
three countries indicated that the lack of top management commitment and poor 
understanding of requirements are the most likely causes of software project failures. 

One of the most prevalent reasons for software cost overruns and schedule delays is 
the lack of user involvement in the planning of the system. The cost of finding a 
problem in the early phases of requirements specification, conceptual design, and test 
planning is relatively low. If discipline specialists and users are not involved in these 
early phases, problems may not be discovered until late in the test and integration 
phases when they can be very costly to fix. 

The concurrent engineering approach will minimize rework costs and schedule 
delays. It will also enhance the safety, performance, and acceptance of the software. 

Ref: Finding #37 

The Data Encryption System (DES) is widely used for encrypting data in a variety of 
Government and commercial activities. NASA made the decision several years ago 
to use DES for encrypting the command uplinks to the ISS, with the expectation of 
upgrading the security system before the end of the station lifetime. DES is certified 
by the National Institute of Standards and Technology (NIST) and is based on the 
use of a 56-bit encryption code. The present certification of DES as an acceptable 
encryption code expired in 1998. It has been expected that NIST will recertify it for 
4 or 5 more years and then replace it with an advanced encryption system. 

For the past 2 years, there has been an organized effort to break the DES code. In 
July 1998, DES was broken with a personal computer and approximately $250,000 of 
special purpose hardware in just 56 hours. The code cracking recovered one DES key 
and deciphered a simple, one-sentence message in English. A complete description 
of the technology, including all code, all circuit diagrams, the chip source code, and 
the system architecture, is available in a newly released book on sale for less than 
$30. Whether or not NIST will recertify DES, in light of the recent success at break- 
ing the DES code, will not be known until after the finalization of this annual report. 

There is a more secure encryption system, called Triple DES, which uses three dif- 
ferent 56-bit keys. NASA did not use this initially because this code is under export 
control and there was concern about difficulties in getting permission for the inter- 
national partners to use this code. Over the past year, NASA has made plans to use 
triple DES with the Ku-band antenna when it is put in place for use with the 
Japanese module. The Ku-band antenna will be available when the U.S. Lab module 
is attached. Although they do not yet have permission to use Triple DES with Japan, 
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the National Security Agency (NSA) has recently indicated that it will not object 
to the issuance of an export license. While NSA does not issue the license, its decb 
sion to not object should help NASA secure an export license not only for Japan but 
for other international partners as well. There would be obvious security benefits 
from incorporating the encryption protocol as soon as possible. NASA is working on 
the upgrade of the current S-band system :n parallel with the deployment of Triple 
DES in the KuTand system, and negotiations on the use of Triple DES are ongoing 
with both the Japanese and European space agencies, 

NASA has some protections for the uplink in addition to that offered by the DES, 
including command signal string authentication and the use of NSA^generated 
encryption keys. First, anyone trying to break the DES code would have to capture 
transmitted sequences and know that they were uplink commands. One would have 
to address the fact that the ISS would only be in contact with a specific location on 
Earth about 10 minutes of each orbit. One would have to know the command format, 
and one would have to address timing constraints on the use of command uplinks. 
Nevertheless, while not openly advertised, none of this information is classified, and 
a determined adversary could probably obtain this information. 

At present, NASA has 4,000 keys availab e for each of the three main ISS uplink 
functions. They could change one key a day for 1 1 years, or one key per hour for a 
year and a half, while they are developing and installing a more secure system. 
However, this alone is unlikely to make breaking the DES code sufficiently difficult 
when one considers the probabilistic nature of the search. 

In view of the recent event, it must now be assumed that, if there is the potential for 
a credible threat to ISS uplink, the DES encryption scheme, regardless of when keys 
are changed, is not going to provide adequ; te protection. A major question, then, is 
whether or not there is a credible threat* ning group. NASA has said it receives 
formal direct threat reports from NSA annually, although there has not been one 
since the breaking of DES in July. Informal reports are relayed to NASA as new 
threats arise, as often as daily. NASA has aid it knows of no explicit threat to the 
ISS uplink at the present time. 

There has not yet been a careful, detailed analysis of the degree of protection the 
system has under the new circumstances of he breaking of DES. It would be useful to 
know how much protection really is availa )le when all of the factors are taken into 
account. Indeed, for something of the value md safety intensity of the ISS, one should 
have a precise analysis of its vulnerability. NSA should be able to perform this kind of 
analysis for NASA. NASA is urged to worl with NSA to obtain such an analysis. 

The time to make an update to the security system is significant. Should a threat arise 
after the launch of the initial ISS components when it is likely to be more visible in 
the world news, it is unlikely NASA could respond in a timely manner. Because there 
was a belief earlier that the risk warranted the use of a secure encryption system and 




there are many attempted break-ins to the NASA computer systems each month, the 
Panel believes that there is still reason to protect the command system. In addition, 
over the extended life of the ISS, it may well be important to offer NASA’s payload 
customers the use of a secure encryption system to protect their uploaded and down- 
loaded data. It therefore seems reasonable for NASA to upgrade the data uplink 
security system as soon as possible and to consider installing downlink security as well. 
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Appendix B 

NASA RESPONSE TO 1097 ANNUAL REPORT 

Summary 

NASA responded on September 14, 1998, to the “Findings and Recommendations” 
from the Annual Report for 1 997* NASA’s response to each report item is categorized 
by the Panel as “open, continuing, or closed.” Open items are those on which the 
Panel differs with the NASA response in one or more respects. They are typically 
addressed by a new finding, recommendation, or observation in this report. 
Continuing items involve concerns that are an inherent part of NASA operations or 
have not progressed sufficiently to permit a final determination by the Panel. These 
will remain a focus of the Panel’s activities during 1999. Items considered answered 
adequately are deemed closed. 

Based on the Panel’s review of the NASA response and the information gathered 
during the 1998 period, the status of the recommendations made in the Annua/ 
Report for 1997 is presented on the following page. 
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Recommendation 


No. 

Subject 

Status 

la 

NASA and United Space Alliances (USA’s) reaffirmation of safety 
before schedule before cost 

Continuing 

lb 

NASA’s development of training and carter paths leading to 
qualification for senior NASA Space Shuttle management positions 

Continuing 

lc 

NASA’s continued commitment that a trained and qualified 
Government personnel presence is maintained on the work floor 

Open 

Id 

NASA and USA’s continued search, devtlopment, test, and 
establishment of operations and processing metrics 

Continuing 

2 

KSC’s expansion of structured surveillance and their development of 
valid and reliable metrics 

Open 

3 

Certification of Self-Contained Atmospheric Protective Ensemble 
(SCAPE) personnel 

Closed 

4 

Cross-training of NASA and USA personnel 

Open 

5 

Downsizing of personnel and the reduction of Government Mandatory 
Inspection Points (GMIPs) and NASA safety inspections 

Continuing 

6 

Adherence to Super Light Weight Tank iSLWT) manufacturing and 
quality control procedures 

Closed 

7 

SLWT design requirements 

Continuing 

8 

Test intervals for flight support motor (FS M) static test firings 

Closed 

9 

Restoration and upgrading of line-replactable units 

Continuing 

10 

Continuation of task management integration of the formerly 
separate logistics contracts 

Closed 

11 

Increased cannibalization rates 

Open 

12 

Readiness of 1SS assemblies prior to shipment 

Closed 

13 

Continued examination of the ShuttleA ir program for ISS benefits 

Closed 

14 

ISS crew radiation exposure levels 

Open 

15 

Review, finalize, and document Caution and Warning (C&W) system 
design requirements 

Continuing 

16 

Revaluation of the achievable ISS softwe re development and test 
schedule 

Continuing 

17 

Importance of maintaining software deve lopment tools 

Open 

18 

Upgrading the ISS computer system 

Open 

19 

Adequate Independent Verification and /alidation (IV&.V) funding 
for the Checkout and Launch Control S' stem (CLCS) 

Closed 


National Aeronautics and 
Space Administration 

Office of the Administrator 

Washington, DC 20546-0001 



Mr, Richard D. Blomberg 
Chairman 

Aerospace Safety Advisory Panel 
1010 Summer Street 
Stamford, CT 06905-5503 


SEP 1 4 1998 


Dear Mr, Blomberg: 

In accordance with your introductory letter dated 
February 1998 in the Aerospace Safety Advisory Panel (ASAP) 
Annual Report, enclosed is NASA ' s detailed response to 
Section II, "Findings and Recommendations . * 

The ASAP ' s efforts in assisting NASA to maintain the 
highest possible safety standards are commendable. Your 
recommendations are highly regarded and continue to play an 
important role in risk reduction in NASA programs. 

We thank you and your Panel members for your valuable 
contributions. ASAP recommendations receive the full 
attention of NASA senior management. In particular, I 
expect that NASA's Office of Safety and Mission Assurance 
will track resolution of these issues as part of their role 
in independent assessment. 

We welcome the continuance of this beneficial working 
relationship with the Panel. 



Enclosure 
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1998 

AEROSPACE SAFETY 
ADVISORY PANEL REPORT 

Findings, Recommendations, and Responses 

A. SPACE SHUTTLE PROGRAM 

Operations/Processing 

Finding #1 

Operations and processing in accordance with the Space Flight Operations Contract 
(SFOC) have been satisfactory. Nevertheless, lingering concerns include: the danger 
of not keeping foremost the overarching goal of safety before schedule before cost; 
the tendency in a success-oriented environment to overlook the need for continued 
fostering of frank and open discussion; the press of budget inhibiting the mainte- 
nance of a well-trained NASA presence on the work floor; and the difficulty of a 
continued cooperative search for the mcst meaningful measures of operations and 
processing effectiveness. 

Recommendation #la 

Both NASA and the Space Flight Operations Contract’s (SFOC’s) contractor, 
United Space Alliance (USA), should reaffirm at frequent intervals the dedication 
to safety before schedule before cost. 

Response 

The Space Shuttle Program concurs with I he ASAP affirmation that safety is our first 
priority. The potential for safety impacts is a result of restructuring and downsizing 
are recognized by NASA at every level. From the Administrator down there is the 
communication of and the commitment to the policy that safety is the most impor- 
tant factor to be considered in our execution of the program and that restructuring 
and downsizing efforts are to recognize th s policy and solicit and support a zero tol- 
erance position for safety impacts. The restructuring efforts across the Program in 
pursuit of efficiencies which might allow downsizing of the workforce consistently 
stress that such efficiencies must be enabh d by identification and implementation of 
better ways to accomplish the necessary work, or the unanimous agreement that the 
work is no longer necessary, but that in e ther case that the safety of the operations 
are preserved. 

In the case of the restructuring and downsizing enabled by the SFOC transition of 
some responsibility and tasks to the contractor, the transition plans for these 
processes and tasks specifically address the safety implications of the transition. 
Additionally, the Program has required the NASA Safety and Mission Assurance 



(S&MA) organizations to review and concur on the transition plans as an added 
assurance. Other Program downsizing efforts have similar emphasis embedded in the 
definition and implementation of their restructuring, and the S&MA organizations 
are similarly committed as a normal function of their institutional and programmatic 
oversight to assure this focus is not compromised. 

Additionally, the Program priorities of 1) fly safely, 2) meet the manifest, 3) improve 
mission supportability, and 4) reduce cost are incorporated into almost every facet of 
planning and communication within both the NASA and contractor execution of 
the Program. Besides the continuous presentation of these priorities in employee 
awareness media, the Program highlights their relative order in the formal consider- 
ation of design and/or process changes being considered by the various Program 
control boards. Additionally, these priorities are the focus point for most of the 
Program management forums such as the Program Management Reviews and SFOC 
Contract Management Reviews (CMRs). They are specified as the basis for the 
Program Strategic Plan, as well as the SFOC goals and objectives used by the con- 
tractor and NASA to manage and monitor the success of the SFOC. Finally, these 
priorities are embedded in the SFOC award fee process (which provides for four 
formal reviews each year). Specifically, the award fee criteria provide for both safety 
and overall performance gates which, if not met by the contractor, would result in 
loss of any potential cost reduction share by the contractor. 

In summary, NASA and all of the contractors supporting the Space Shuttle Program 
have always been and remain committed to assuring that safety is of the highest pri- 
ority in every facet of the Program operation. While downsizing does increase the 
challenge of management to execute a successful Program, process changes, design 
modifications, employee skills maintenance, and reorganizations are all part of the 
management challenges to be faced and resolved, and maintenance of the high level 
of attention to safety in resolving these challenges is recognized by NASA and the 
contractors alike as not being subject to compromise. 

Recommendation #1 b 

NASA should develop and promulgate training and career paths leading to qualifi- 
cation for senior NASA Space Shuttle management positions. 

Response 

While it is true that the roles for NASA management and technical personnel are 
being reduced in number and reshaped to focus on the critical areas of anomalies and 
changes, these roles and the ongoing role of assessing the contractor’s performance 
against the contract and Program requirements should provide a continued source of 
trained and capable future NASA senior managers. NASA has an active commit- 
ment to development of the skills for senior managers for all functional areas of the 
Agency, and Space Shuttle Program senior managers are generally products of both 
their in-line experiences as well as these career development programs. It is antici- 
pated at this time that the roles for NASA personnel and the career development 
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programs which have served NASA well to this point will be sufficient to assure a 
continuation of highly qualified and capable senior managers in the future. Given 
the nature of the still evolving definition of the NASA and prime contractor roles 
and responsibilities for the SFOC operational model, it is reasonable to provide spe- 
cial attention to this concern, and the Program will ensure that specific 
consideration is given to this concern in the transition plans being developed and 
implemented by the functional and institutional organizations across the Program. 

Recommendation #lc 

NASA should continue to ensure that a :rained and qualified Government person- 
nel presence is maintained on the work floor. 

Response 

NASA/KSC has maintained a physical presence on the work floor since the begin- 
ning of the Shuttle Processing Contract and will continue this presence for SFOC, 
Payload Ground Operations Contract, ard Base Operations Contract. NASA engb 
neering, operations, safety, and quality personnel maintain a surveillance and audit 
presence of overall operations for insight purposes and are formally involved for 
selected tasks being performed. Presence on the floor monitoring hazardous or safety 
critical operations has been maintained through the transition to performance based 
contracting and will be maintained in the future. The frequency and depth of the 
insight and presence may be adjusted as justified by the results of the contractors per- 
formance, but the value of these checks and balances has long been recognized by 
NASA and will be maintained. To a lesser degree, this same floor presence is exe- 
cuted at production sites through Resident Office presence and periodic audit and 
surveillance activities by NASA Center personnel. 

While there is a focused initiative to ml nimize Government mandatory inspection 
points (GMIP’s) across the Program, it b mutually recognized by NASA and USA 
that the criticality of some checks and balances in critical processes demands that 
some small percentage (10-15 percent) vill be maintained on the production and 
processing floors. This presence also supports the desired training and qualification 
needs for NASA to remain a smart customer. Finally, there are functional roles antic- 
ipated for continued NASA participatior , such as flight controllers, astronauts, and 
launch directors which will also provide i significant avenue for NASA skills main- 
tenance in the long-term management m xlel. 

R e commendation #ld 

NASA and USA should continue to sear :h for, develop, test, and establish the most 
meaningful measures of operations and piocessing effectiveness possible. 

Response 

Both NASA and USA recognize the value of meaningful measures of the operational 
and processing effectiveness for the Program and continually strive to evolve and 
improve on the measures currently in place. The SFOC Performance Measurement 



System (PMS) has been a significant development project since the beginning of the 
contract, continues to take shape as the primary repository for the performance met- 
rics which provide management insight into the cost, and technical performance 
across the complete contract. Once the system is complete and populated with viable 
metrics, NASA will validate the system. The goal is to complete the validation by 
the fall of 1998. Key metrics are reviewed quarterly at the SFOC CMR, and individ- 
ual functional areas such as flight operations and ground processing use these on a 
continual basis for their management execution and insight. Additional measures are 
continually developed at the Program level and within individual functional areas to 
enhance the understanding of performance trends, and when proven to be effective 
management tools, these metrics roll into the PMS and/or other forums and products 
used to manage the Program. 

Finding #2 

The Kennedy Space Center (KSC) has been successfully phasing in the structured 
surveillance process for safety and quality for some time. The development of metrics 
using structured surveillance information has lagged data collection. 

Recommendation #2 

KSC should continue to expand the use of structured surveillance and to focus effort 
on the development of valid and reliable metrics to assess program performance from 
structured surveillance results. 

Response 

We concur. The development of reliable metrics with which to measure performance 
of the SFOC in all areas including safety and quality is progressing at KSC. There are 
several examples of this. 

At KSC, NASA Safety has developed a data base for the Space Shuttle Program, 
revised its surveillance approach, and developed a method by which proper mea- 
surements can be evaluated and analyzed in determining safety program management 
effectiveness and contractual statement of work compliance. These new metrics will 
enable NASA Safety to more effectively measure contractor performance. 

The Quality Surveillance Record data base is currently being modified to clarify the 
method by which deficiencies and observations are counted and to better define fail- 
ure codes and other data collected. These changes will increase the reliability of the 
data used to assess program performance and will be implemented in early July. In the 
interim, the existing data base has been modified and focuses on surveillance data 
collection for tasks which GMIP’s were deleted through the GMIP reduction efforts. 

KSC has developed an expanded surveillance system that will provide extensive 
insight into the contractor’s overall operation by process analysis. The process analy- 
sis program was initiated in October 1997, and there are presently 1 1 Quality Process 
Analysts working the pilot program at KSC. This system will provide added insight 
into the contractor’s processes, procedures, and policies. 
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Finding #3 

NASA Safety and Mission Assurance (S&MA) auditors at KSC overseeing opera- 
tions requiring Self-Contained Atmospheric Protective Ensemble (SCAPE) are not 
certified for SCAPE. 

Recommendation #3 

In order to be in a position to conduct valid safety and quality audits of SCAPE 
operations, NASA should ensure that peisonnel involved are certified so that, when 
necessary, they can observe the tasks whi e they are performed. 

Response 

The Space Shuttle Program concurs that safety and quality audits of SCAPE opera- 
tions be performed. However, KSC’s pcsition is that NASA’s safety and quality 
personnel monitoring SCAPE tasks will not be exposed to the additional risk of 
SCAPE operations as personnel can accomplish monitoring tasks by observing and 
communicating through the audio and video capabilities of the Operational 
Intercommunication System and operational television (OTV). All SCAPE tasks are 
conducted on recorded communications channels that are monitored in the control 
room, and the majority of tasks are observable on the OTV system. NASA quality 
and safety personnel have performed th )se audits for several years without being 
SCAPE certified. 

Finding #4 

To compensate for skills deficiencies related to staff departures from KSC, both 
NASA and USA are making extensive use of cross-training of personnel, both tech- 
nicians and engineers. Individuals who nave been cross-trained also should have 
recent “hands-on” experience before they undertake a cross-trained task. 

Recommendation #4 

NASA and USA should develop and use valid and reliable measures of the readiness 
of personnel to take on tasks for which the i have been trained but on which they have 
only limited or episodic experience. The cross-training program could include a regu- 
larly scheduled rotation of duties so that the multiply trained individual has the 
opportunity to employ all of the acquired si dlls and knowledge at appropriate intervals. 

Response 

NASA is in full agreement with the Pane ’s position that individuals who have been 
cross-trained also should have recent har ds-on experience before performing tasks. 
The combined NASA/USA training and certification plan identifies those skills 
that require hands-on training as part of the certification process. Personnel selected 
for cross-training are required to be certified to perform other jobs in the same 
family of skills (i.e., mechanical systems, £ vionics systems, electrical distribution sys- 
tems). With this knowledge base, those identified for cross-training will be required 
to meet the same training and performance requirements established for the given 
task. Performance is measured to verify that the individual has obtained the stated 


objectives of the instructional tool being used. In the case of hands-on training, the 
employee is required to demonstrate 100% command of the task being performed. 
The certification process is controlled by the KSC Certification Board, which oper- 
ates under approved certification procedures. The Certification Board, chaired 
by the USA S&MA Director at KSC, approves and implements certification/re- 
certification requirements. 

Finding #5 

The reduction of Government Mandatory Inspection Points (GMIPs) at KSC has 
significantly lagged the downsizing of NASA quality personnel responsible for pro- 
cessing these GMIPs. This has resulted in an expanded workload among remaining 
NASA quality inspectors and made it more difficult to conduct analyses needed to 
identify further GMIP reductions. There has been a similar reduction of NASA 
safety inspectors and engineers at KSC without a commensurate reduction in over- 
sight requirements while, at the same time, the addition of new safety audit or insight 
responsibilities has taken place. 

Recommendation #5 

Any downsizing of personnel by both NASA and USA should be preceded by the 
reduction of commensurate workload associated with Space Shuttle processing, such 
as reduction of GMIPs and NASA safety inspections. 

Response 

NASA concurs with the recommended approach of reducing the workload in Space 
Shuttle processing before proceeding with downsizing NASA and USA personnel; 
however, we have not been as successful in this area as desired. In the downsizing 
effort implemented in February 1998, USA experienced an unexpectedly high level 
of voluntary attrition in certain critical functions — an outcome that was predicted 
by ASAP members and others. Although USA experienced shortages of critical 
skills and staffing to minimum levels for short periods of time in selected areas, 
USA and NASA worked together to overcome these deficits and assure that the 
scheduled missions through STS-91 were safely executed. This was done by a com- 
bination of launch schedule relief, back-filling USA shortages with NASA 
expertise, and re-hiring technical expertise to train and certify USA staff, thus 
eliminating shortages in critical skills. Evaluation of GMIPs for potential elimina- 
tion by process engineering and quality engineering staff continues. It is estimated 
that approximately 6,000 of the original 22,000 GMIPs will remain in place at the 
end of this effort. This is a level assessed as commensurate with the current NASA 
quality inspection workforce. 

NASA Headquarters Office of Safety and Mission Assurance (OSMA) continues to 
evaluate the situation at KSC regarding NASA and USA workforce reductions by 
assessing process efficiencies and workload indicators. Indicators of process effective- 
ness include overtime rates and first-time quality rates. Although the efforts are not 
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yet complete, OSMA anticipates that as GMIPs are reduced overtime rates for 
NASA quality inspections will drop. Additionally, if the development of process effi- 
ciency initiatives by USA are effective, then, when implemented, OSMA anticipates 
that USA engineering and technician ovt rtime rates will drop and first^time quality 
rates, based on NASA surveillance sampl ng, will increase. 


External Tank (ET) 

Finding #6 

The Super Light Weight Tank (SLWT) has completed its design certification review, 
and proof tests on the first tank have been satisfactorily passed. The only remaining 
test to complete certification on the SLWT is the cryogenic loading test that will be 
run on the first production tank on the launch pad. The diligent attention that has 
been given to quality control, particularly to material inspection and weld integrity, 
has made this program successful. 

Recommendation #6 

NASA should ensure that the current manufacturing and quality control procedures 
continue to be rigidly adhered to and conscientiously followed in production. 

Response 

NASA concurs with this recommendation. MSFC and Lockheed Martin Michoud Space 
Systems (LMMSS), the External Tank prime contractor, periodically perform a NASA 
Engineering and Quality Audit (NEQA) which focuses on both the processes and the 
flight hardware. The audit is conducted by experienced MSFC and LMMSS technical 
and management personnel and the operators and inspectors that actually utilise those 
processes. LMMSS also performs internal and supplier audits throughout the year. In 
addition, on-site MSFC Science and Engineering, Safety and Mission Assurance, and 
DCMC personnel provide continuous insight and guidance through surveillance and 
limited oversight activities. Finally, adherence to manufacturing and quality control pro- 
cedures is one of the primary focuses of the on-site government personnel. 

Finding #7 

The design requirements for the SLWT include operating with a maximum Space 
Shuttle Main Engine (SSME) power of only 106%, even at abort conditions. The 
Space Shuttle program has approved a baseline plan to examine the possibility of cer- 
tifying the Space Shuttle for intact aborts at a 109% SSME power setting. 

Recommendation #7 

NASA should complete its evaluation of a 109% power setting for intact aborts as 
soon as practicable and reevaluate the ability of the SLWT to accommodate this 
higher power setting. 

Response 

NASA concurs with the ASAP recommendation. Specific evaluations with regards 
to orbiter and SLWT have already been completed or are near completion. The 
Block II Space Shuttle main engine (SSME) certification program will provide the 
capability for intact abort at 109 percent power level. This certification program is 
planned to be complete by October 1, 1998. A change request to baseline the 
109 percent intact abort loads and thermal environments shall be released by the end 
of 1998 following completion of Block II SSME 109 percent certification. 
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Reusable Solid Rocket Motor (RSRM) 

Finding #8 

Obsolescence changes to the RSRM processes, materials, and hardware are continu- 
ous because of changing regulations and c ther issues impacting RSRM suppliers. It is 
extremely prudent to qualify all changes n timely, large-scale Flight Support Motor 
(FSM) firings prior to produce/ship/fly. NASA has recently reverted from its planned 
12-month FSM firing interval to tests on 18-month intervals. 

Recommendation #8 

Potential safety risks outweigh the small amount of money that might be saved by 
scheduling the FSM motor tests at 18- month intervals rather than 12 months. 
NASA should realistically reassess the test intervals for FSM static test firings to 
ensure that they are sufficiently frequent o qualify, prior to motor flight, the contin- 
uing large number of materials, process, and hardware changes. 

Response 

Evaluation of all known reusable solic rocket motor (RSRM) future material, 
process, and hardware changes (by NASA and Thiokol) has confirmed no safety risk 
impact resulting from FSM static tests every 18 months, in lieu of every 12 months. 
The RSRM Project goal to “include all c nanges in a static test prior to flight incor- 
poration” has not changed, and any exceptions will continue to be approved by the 
Space Shuttle Program Manager before f ight incorporation. If a change is planned 
in the future wherein an 18-month FSM static test frequency is insufficient to sup- 
port qualification prior to motor fligh , program funding requirements will be 
considered to accelerate an FSM static est to ensure no increased program flight 
safety risk. 




Logistics 
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Finding #9 

Support of the Space Shuttle fleet with operational spares has been maintained by 
the effective efforts of the logistics function. While spares support has been adequate 
for the current flight rate, any increase in flight rate might not be supportable. 

Recommendation #9 

Although NASA has established programs for dealing with suppliers and bringing 
additional component overhaul “in house,” efforts in these areas need to be contin- 
uously reexamined to speed up the restoration and upgrading of line-replaceable 
units. Such efforts are especially needed to eliminate “dead” time while units are 
awaiting restoration. 

Response 

The Space Shuttle Program concurs with the ASAP concerns for the availability of 
line replaceable units (LRU’s). Logistics monitors LRU spares posture through the 
probability of sufficiency calculations within the LRU data system. This system can 
be programmed to determine spares requirements for various flight rates. At this 
time, an appreciable increase in flight rate would be required to jeopardize support- 
ing the Space Shuttle Program with most of the current LRUs. 

The Program has been proactive in upgrading LRU’s where the most pressing fleet 
support concerns exist. The following upgrades are in progress: 

1. The air data transducer assembly is being replaced by the advanced air data 
transducers. 

2. The master events controller is being replaced by the advanced master events 
controller. 

3. The Global Positioning System is being installed in the orbiters at the orbiter 
major modification. This could potentially eliminate a number of LRU’s in the 
displays and controls system and tactical air navigation system. 

4. New shop replaceable units were purchased to repair the Microwave Scanning 
Beam Land Station decoders. This decreases turnaround time and increases 
reliability of the units. 

5. Solid state recorders are being considered as replacements for the operations 
and payload recorders. This effort is presently in the design definition phase. 

Additionally, through the use of industrial engineering principles and work teams, 
Logistics has taken action to reduce the NASA Shuttle Logistics Depot (NSLD) 
backlog and increase output for fiscal year 1998. To date, backlog has decreased 8 per- 
cent since October 1 by increasing output. These efforts are aimed at providing better 
support at the current flight rate but are also the type of efforts that will allow higher 
flight rates in the future. 
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Finding #10 

Transition and development of the logisti zs tasks for the orbiter and its ground oper- 
ations under the SFOC are proceeding ef iciently and according to plan. 

Recommendation #10 

NASA and USA should continue the task of management integration of the for- 
merly separate logistics contracts and retain and expand the roles of the experienced 
logistics specialists therein. 

Response 

The Space Shuttle Program concurs with the ASAP philosophy of logistics integra- 
tion. Integrated Logistics has been successful in integrating Ground Logistics and 
more recently, Flight Operations Logistics with Orbiter Logistics insight. As new ele- 
ments are integrated within USA, the sharing of new techniques and best in class 
practices is occurring. Logistics is recognized as a key member of both the NASA and 
contractor teams; their input is actively sought on key decisions, and they are mem- 
bers of key decision-making boards and panels. 

Finding #11 

As reported last year, long-term projections are still suggesting increasing cannibal- 
ization rates, increasing component repair turnaround times, and loss of repair 
capability for the Space Shuttle logisti :s programs. If the present trend is not 
arrested, support difficulties may arise in the next 3 or 4 years. 

Recommendation #11 

NASA and USA should reexamine and take action to reverse the more worrying 
trends highlighted by the statistical trend data. 

Response 

The Space Shuttle Program has recognize J the concerns for long-term supportability 
and is proactively pursuing improvements. Cannibalizations continue to be closely 
monitored and are well within limits. Th^re have been several concerns during this 
past year (seals and cryo heater controller; ) that are requiring the adjustment of spar- 
ing levels. The Logistics organization is aggressively pursuing a solution to specific 
problems as well as pursuing innovations to keep the rate below the standard. 

As mentioned in the response to Finding 5*9, the NSLD backlog is now decreasing as 
a result of USA action. This should ultimately reduce the repair turn around time for 
hardware although short term increases can be expected. Other initiatives such as 
the replacement of unserviceable test equipment at vendors are also in progress. 

Logistics and Engineering are developing :ommon tools to integrate upgrade actions, 
resolve supportability issues, and mitigate the loss of repair capability. The Problem 
Resolution Teams have increased the interfaces with Logistics, Engineering, and 
management to ensure a proactive and integrated effort in identifying problem areas 
and identifying solutions. Numerous initi itives are under way. 




Finally, NASA has funded through Space Shuttle upgrades the prototyping of a new 
expert logistics system which shows promise in ranking issues according to severity. 
This data might then be used to assure that limited funding available is used as ea> 
nomically and wisely as possible in order to minimize risk in the most vulnerable areas. 
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B. INTERNATIONAL SPACE STATION (ISS) PROGRAM 

Finding #12 

Node #1 was shipped to KSC before completion, and it is planned or anticipated that 
other ISS hardware will be shipped before qualification tests are completed. This dis- 
rupts the desirable continuity of effort and can lead to safety problems. 

Recommendation #12 

NASA should assure that ISS assemblies si ipped before completion of the manufac- 
turing, testing, and qualification processes have been carefully scrutinized to make 
sure that no safety-related steps are subverted. 

Response 

The generation, implementation and traccing of ISS assembly, checkout and test 
requirements is provided by engineering, configuration management and safety and 
mission assurance processes that are in plat e and actively monitored. Should a deci- 
sion be made to transfer flight articles to a different location for completion of 
planned assembly and checkout activities, these processes ensure an accurate status 
of accomplished and traveled work is known and documented. In addition, a pre- 
determined set of criteria identifies the set of minimum essential requirements to be 
accomplished prior to shipment. 

All ISS assemblies undergo a rigorous predelivery review prior to shipment to KSC. 
This high-level review is attended by Senior NASA and contractor representatives 
from all the major functional disciplines involved in the specific assembly in ques- 
tion, including Engineering, Quality, Safet/, Hardware Integration Office, and KSC 
Processing. Open work items that are candidates for completion at KSC are specifi- 
cally addressed, validated and accepted by all in attendance prior to being forwarded 
for integration into the contractors existing work plan for KSC. 

No work practices or safety related practice ; are compromised by this process. Boeing 
remains completely accountable for work eompletion and providing complete and 
fully tested ISS components to NASA rt gardless of the physical location of the 
assembly process. 

Finding #13 

The ISS Phase I Shuttle-Mir program has reaffirmed what was learned on Skylab: 
that a manned space station can be surpr singly resilient in emergency situations. 
Much has been learned from the operatiors on Mir to date and much more may be 
learned from continued analysis of joint operations on Mir. 

Recommendation #13 

The ISS team should continue to examire the Shuttle-Mir program carefully for 
examples from which ISS operations can benefit and to provide policies and proce- 
dures to implement effective action should similar events occur on the ISS. The 




effort should be expanded beyond Mir to focus as well on possible weaknesses in the 
ISS design and operations. ISS should assemble a special team including, persons 
with system-level perspectives as well as with design, operations, and human factors 
experience, to address these issues. 

Response 

The ISS Program (ISSP) has benefited greatly from the Phase I Program experiences 
in the areas of operational feasibility and validation, procedures development and 
logistics manifesting, and in some cases, hardware modifications. 

A more rigorous process for evaluating Shuttle/Mir lessons learned has been imple- 
mented and will assure that ISS realizes maximum benefit from Shuttle/Mir lessons 
learned. Phase I management screens and prioritizes lessons learned from each 
Shuttle/Mir flight increment to document significant and applicable lessons to ISS. 
These lessons are thoroughly reviewed by ISS Lessons Learned Screening Panel. For 
each lesson, actionee(s) are assigned to analyze ISS applicability and possible imple- 
mentation or rationale for nonimplementation. The Lessons Learned Screening 
Panel and ISSP management will assure that the proposed implementation is appro- 
priate. This screening process includes representatives from numerous organizations 
to ensure that all these issues are adequately addressed. This same process is used to 
incorporate lessons learned which are discovered within the ISS Program as well. 
(Note: Process is depicted in the accompanying figure.) 

• Identify Significant Lessons from Phase I 

- Each Phase I working group to identify key lessons 

- Eliminate duplication and focus on high impact lessons 

• Front End Analysis 

- Determine root causes and add appropriate level of detail 

• Screen Lessons 

- ISS Lessons Learned Screening Panel has been established to prioritize 
and categorize lessons 

- Determine responsible person/organization to respond 

• Document and Track 

- Enter lessons into database and track disposition 

• Decompose, Analyze, and Disposition Lessons 

- Systems Engineering used to determine applicability and impacts 

- Disposition: implement, partially implement, or no practical implementation 

Another specific example of Phase I lessons learned implementation: Unplanned 
events on Mir resulted in the requirement to provide late stowage of items on the 
Shuttle. The requirement to support this activity has reinforced the importance of 
building flexibility into our ground processing capabilities and operational planning 
for the ISS Phase 2 resupply missions. In order to accommodate the potential for 
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similar requirements during Phase 2, we have designed and are in the process of 
developing support equipment which will permit contingency access to the Multi' 
Purpose Logistics Module (MPLM) after it is installed in the orbiter at the Pad. This 
will enhance our ability to react to changes which require the addition of items late 
in the processing flow. 

The Manager, Phase 1 Program, has been directed by the Lead Center director to per' 
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display development, audio interfaces, flight rules/procedures, and IP development 
areas. The C&W review resulted in a total of 35 issues requiring resolution. These 
issues ranged from editorial comments against the C&W Description Document to 
the identification of Russian audio interface concerns, Japanese Experiment Module 
C&W Panel Latencies, requirements modifications, and needed display design 
changes. The finalized C&W system requirements are documented and controlled 
within the formal ISS configuration mansged specification hierarchy, and any mod' 
ification, waiver, or deviation requires offizial Program direction. 

The CWSIT, which has oversight and extensive involvement with the class 1/emer' 
gency procedures, has developed and retains ownership of the classification 
guidelines/requirements, and maintains control of these requirements through the 
C&W Working Group. The actual procedure development is assigned to the Mission 
Operations Directorate (MOD), who is responsible for the particular system, under 
the direction of the MOD management and Joint Operations Panel. The CWSIT has 
briefed the development organizations on the general philosophy of the class 1/2/3 
procedures required for C&W. In addition, the CWSIT reviews all changes to the 
class 2/3 procedures after they are placed under configuration control. 

The integration of C&W event response procedures is ensured across the program by 
the CWSIT and its constituent MOD, Crew Office, and program subsystem repre' 
sentatives. The situation response and damage control procedures are included in the 
baselined and configuration managed ISS Mission Rules and Procedures. These flight 
procedures are integrated into the overall lisplay development and design modifica' 
tion processes currently in work. 

The Boeing developed MatriX'X C&W s mulator has been brought ondine and is 
providing simulation support to the program as well as providing a data driven dis- 
play assessment environment. This simu ator has already provided much needed 
integration support to requirements and design development and is now geared 
towards supporting procedure development and crew training. The simulator has 
been instrumental in the performance of ntegrated crew reviews of both flight 2A 
procedures and display architectures. The use of this simulator in conjunction with 
the Flight Crew Training Division’s Part T isk Trainers has contributed to the deveL 
opment of an integrated, productive, and ‘afe C&W system. 


C. COMPUTER HARDWARE/SOFTWARE 

Finding #16 

The ISS software development schedule is almost impossibly tight. If something else 
does not cause a further delay in ISS deployment, software development may very 
well do so. The decision this year to add integrated testing of some modules at KSC 
is a very positive step for safety. However, there is no room in the schedule for 
required changes that may be discovered during this testing. 

Recommendation #16 

NASA should realistically reevaluate the achievable ISS software development and 
test schedule and be willing to delay the ISS deployment if necessary rather than 
potentially sacrificing safety. 

Response 

The Program has established an aggressive activity to integrate developer schedules with 
need dates (including training as well as test). Schedules are difficult but proving to be 
achievable. Any disconnect, whether it is schedule or content, is tracked and worked 
through the Program’s formal decision process on a daily basis. Staffing is reviewed weekly 
and will be sustained to meet commitments. Additional independent verification and 
validation and software assurance support has been added. The program is firmly com' 
mitted to our test plans and will make the appropriate schedule adjustments to maintain 
these plans. The recently approved Revision D to the ISS assembly sequence provides 
additional schedule flexibility to accomplish all testing and software activities. Finally, the 
program will not commit to flight until the software has been adequately tested. 

Finding #17 

NASA does not yet have adequate plans for the long'term maintenance of the soft' 
ware development tools being used to produce the ISS software. 

Recommendation #17 

NASA should recognize the importance of maintaining its software development 
tools, plan now for how these are to be maintained over a period of decades, and pro' 
vide adequate funding to support this activity. 

Response 

Provision for support of software development tools is provided in the ISS Sustaining 
Engineering Plan. Funding is also provided to maintain Ada compiler license and soft' 
ware support. This includes a clause requiring delivery of source code in the event of a 
provider decision not to support the compiler users at a later date. The GFE software is 
maintained by intenorganizational Technical Task Agreements ( I TA’s) which will be 
managed by the same Sustaining Engineering organization that is responsible for all ISS 
integrated software maintenance’s and upgrades. Activity is also underway to invests 
gate the impact of upgrading the Ada compiler to a more current design or even to 
consider moving away from Ada to other widely universally supported languages, as a 
part of the ISS Preplanned Production Improvement activity. 
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Finding #18 

The computer system being developed for the 1SS is already at a point where NASA 
should begin planning for upgrading it. The ISS program presently has no plans for 
upgrading the ISS computer system. 

Recommendation #18 

NASA should upgrade the computer system as soon as possible and coordinate the 
upgrade with its solution to the long-term development tool maintenance problem. 

Response 

The Program’s computer system must be v ewed in two parts: 1) the core infrastruc- 
ture; i.e. multiplexers/demultiplexers (MDM’s) and 2) the user interface; i.e. Portable 
Computer System. The core infrastructure maintains the facility environment and 
basic command and control interface. This functionality will require minimal 
growth, but obviously must be capable of being maintained over the period of ISS 
life. The user interface, on the other hand, must be capable of growth to enhance 
productivity as well as be maintainable. With this in mind, the user interface is 
developed using commercially available hardware and software and will be upgraded 
as technology progresses. In fact, one upgiade is already being implemented at 5 A. 
Provisions for maintenance of the compute r system is provided in the ISS Sustaining 
Engineering Plan. Funding is provided to maintain critical skills to support flight and 
ground hardware, including support engineering and touch labor to repair cards and 
provide new spares; this includes maintaining critical facilities. An ISS Pre-Planned 
Program Improvement (P3I) study is under review to evaluate a more current design 
upgrade to the MDM “386” processor in FY 99, to ensure MDM core infrastructure 
sustainability and adequate growth potent al during the ISS lifetime. In addition a 
technical new start for an enhanced mass . torage device for the MDM is also under 
review to improve reliability and storage capability. 

Finding #19 

The Checkout and Launch Control System (CLCS) program at KSC has not been 
provided with funding for Independent Verification and Validation (IV&V) that is 
safety critical for a software effort of this si: e. 

Recommendation #19 

The Checkout and Launch Control System (CLCS) should be provided with ade- 
quate funding for software IV&V. 

Response 

KSC concurs with the ASAP recommenc ation relative to IV&V funding for the 
CLCS Project. A Memorandum of Agreen ent (MOA) was signed on May 5, 1998, 
between the Software IV&V Facility and <SC, for the performance application of 
IV&V techniques and methods to the CLCS software. The scope of this memoran- 
dum will include performing IV&V on selected catastrophic/critical/high risk CLCS 
software components. The selected software components will consist of CLCS system 



software. The specific areas to be analyzed will be system redundancy, command sup- 
port, data distribution and processing, constraint management, and the safing system 
related software. The software related to safing includes the Emergency Safing 
System and those control logic modules associated with safing (some of which may 
reside within application software). The analysis will consist of requirements, design, 
code, and test analysis, as applicable for the life cycle of the software being analyzed. 
The application interfaces with the system software will also be analyzed. In addition, 
the IV&V Facility will perform system level analysis of the system test plan and 
system tests performed along with software engineering and integration analysis of 
the CLCS system as a whole. 

This MOA is effective from May 1, 1998, until September 30, 2000. The work iden- 
tified in this MOA will require a staffing level of about 16 full time equivalents 
(FTEs). This staffing level will be comprised of 15 FTE’s from the IV&V contractor 
located at the 1V&V Facility and at KSC. The remaining one FTE will be a civil ser- 
vice personnel. Staffing at KSC will be comprised of eight contractor FTEs with the 
remainder residing at the Fairmont Facility. 

The Space Shuttle Program has agreed to fund this effort at $4.5M over the life of 
the MOA. 
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Appendix C 

AEROSPACE SAFETY ADVISORY PANE ACTIVITIES 
JANUARY-DECEMBER 1998 

January 

7 Kennedy Space Center, STS-89 Flight Readiness Review (FRR) 

February 

11- 12 Headquarters, Aerospace Safety Advisory Panel Annual Meeting 
19-20 Lockheed Martin “Skunk Works,” Review of the X-33 Program 

25 Pratt and Whitney and Kennedy Space Center, KSC/SFOC Team Visit 

March 

17-19 Kennedy Space Center, KSC/SFOC Team Visit 

23 Johnson Space Center, International Space Station Task Force Meeting 

April 

1-2 Kennedy Space Center, SFOC Contract Discussions and STS-90 Flight 
Readiness Review 

8-9 Ames Research Center, Aeronautics Team Visit 
19-21 Kennedy Space Center, ITV Meeting 

May 

4-5 Kennedy Space Center, Human Factors Workshop 

12- 14 Johnson Space Center, Plenary Session 

19 Kennedy Space Center, Attend STS'91 FRR 
21-22 Orbital Sciences Corporation, X-34 Meeting 

June 

12 Johnson Space Center, United Space Alliance Advisory Board Meeting 
16-18 OEA, Colorado Electric, Ball Aerospace, and Lockheed Martin, Space 
Shuttle Vendor Visits 

23 Ogden, Utah, Space Shuttle Program Manager’s Review 
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July 

8 Dryden Flight Research Center, Attend LASRE FRR 

15-16 Kennedy Space Center, Super Safety Day and Meeting with United Space 
Alliance 

22- 23 Johnson Space Center, Comput ?r Team Visit 

29 Arlington, Virginia, United Space Alliance Advisory Board Meeting 

August 

4 Langley Research Center, Aeronautics Team Visit 

18-19 Kennedy Space Center, KSC Team Visit 
26-27 Marshall Space Flight Center, Plenary Session 

27 Seattle, Washington, John McEonald Accepted the Jack Williams Space 
Logistics Medal 

September 

10-1 1 Johnson Space Center, EVA Te; m Visit 

October 

6-7 Canoga Park, California, Space Shuttle Program Manager’s Review 

9 Lewis Research Center, Attend Turning Goals Into Reality” Conference 

13 Kennedy Space Center, STS-95 FRR 

14 Headquarters, Workforce Issues/ ^act-Finding Meeting 
14-15 Dryden Flight Research Center, Aeronautics Team Visit 
23 Johnson Space Center, ISS Program Readiness Review 

26 Headquarters, Attend AA for CSMA Briefing to Mr. Goldin 

27 Kennedy Space Center, STS~95 LU and Launch 

November 

4 Independent Verification and V ilidation Facility, Fairmont, West Virginia, 

Computer Team Visit 

4-5 Kennedy Space Center, Attend Integrated Logistics Panel Meeting 
12—13 Headquarters, Plenary Session 

1 6 Kennedy Space Center, Comput er Team Visit 
17-18 Ames Research Center, Aeronautics Team Visit 

23- 24 Kennedy Space Center, STS-88 FRR 

December 

1-2 Headquarters, Editorial Commit :ee Meeting 
8 Los Angeles, California, Thiokol Propulsion Supplier Briefing 

17 Headquarters, Editorial Commit :ee Meeting 


